Nov 03 18:37:07.632009 ip-10-0-128-95 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:37:08.027517 ip-10-0-128-95 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:08.027517 ip-10-0-128-95 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:37:08.027517 ip-10-0-128-95 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:08.027517 ip-10-0-128-95 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:08.032271 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032177 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.032271 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032266 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.032271 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032270 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.032271 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032273 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.032271 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032276 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.032271 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032278 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.032271 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032282 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032285 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032288 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032291 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032293 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032296 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032299 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032302 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032305 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032308 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032311 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032313 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032316 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032319 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032321 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032324 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032326 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032329 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032332 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032335 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.032510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032337 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032340 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032343 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032345 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032348 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032351 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032355 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032361 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032364 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032367 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032369 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032373 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032377 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032381 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032386 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032389 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032392 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032395 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032397 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.033035 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032401 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032403 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032406 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032408 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032411 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032414 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032416 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032419 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032422 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032424 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032427 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032430 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032432 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032435 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032437 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032440 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032443 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032445 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032448 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032450 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.033479 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032453 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032455 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032458 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032460 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032463 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032466 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032469 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032472 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032475 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032477 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032480 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032483 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032486 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032488 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032491 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032494 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032496 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032499 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032503 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032506 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.034006 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032508 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032511 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032514 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032516 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032519 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032522 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032524 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032527 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032529 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032532 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032535 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032539 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032541 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032544 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032546 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032549 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032552 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032554 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032557 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.034476 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032560 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.032563 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033381 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033390 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033393 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033396 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033399 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033402 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033405 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033407 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033410 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033413 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033416 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033418 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033421 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033424 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033426 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033429 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033431 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.034949 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033435 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033437 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033440 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033442 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033445 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033448 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033451 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033453 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033456 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033458 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033461 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033463 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033466 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033469 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033472 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033475 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033477 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033480 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033482 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.035410 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033485 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033488 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033490 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033493 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033496 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033499 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033501 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033518 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033521 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033524 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033527 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033529 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033532 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033535 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033539 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033542 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033546 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033549 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.035974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033552 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033555 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033558 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033561 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033564 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033567 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033571 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033574 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033576 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033579 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033582 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033584 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033587 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033589 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033592 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033595 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033598 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033600 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033603 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033605 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.036394 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033608 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033611 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033614 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033616 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033619 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033621 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033624 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033627 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033629 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033631 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033634 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033636 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033639 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033642 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033644 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033649 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033652 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033655 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033658 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033660 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.036910 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033662 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033665 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033668 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033670 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033673 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033675 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033678 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033681 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033684 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033687 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033690 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.033692 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034213 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034219 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034222 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034225 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034228 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034231 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034235 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034239 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.037380 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034243 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034246 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034248 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034251 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034254 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034257 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034260 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034263 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034266 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034269 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034271 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034274 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034276 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034279 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034282 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034284 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034287 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034289 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034292 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.037887 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034295 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034297 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034300 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034302 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034305 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034307 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034310 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034312 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034315 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034318 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034321 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034324 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034326 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034329 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034332 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034335 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034337 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034339 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034342 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034345 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.038335 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034348 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034350 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034353 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034355 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034358 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034361 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034364 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034366 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034369 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034371 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034374 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034377 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034379 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034382 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034384 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034387 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034389 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034392 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034394 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034397 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.038802 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034402 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034405 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034407 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034410 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034412 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034415 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034417 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034420 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034422 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034424 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034427 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034429 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034432 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034435 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034437 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034440 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034442 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034445 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034447 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034450 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.039267 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034453 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034457 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034461 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034464 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034466 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034469 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034472 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034474 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034477 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034480 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034483 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034485 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034493 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034496 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034499 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034502 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034504 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034507 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.034511 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035602 2582 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:37:08.039758 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035611 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035619 2582 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035623 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035628 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035631 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035635 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035639 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035643 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035646 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035649 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035652 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035655 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035658 2582 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035661 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035664 2582 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035667 2582 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035670 2582 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035673 2582 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035676 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035679 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035682 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035685 2582 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035690 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:37:08.040232 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035693 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035709 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035713 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035716 2582 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035720 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035723 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035726 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035729 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035733 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035736 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035739 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035742 2582 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035748 2582 flags.go:64] FLAG: --enable-server="true" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035751 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035756 2582 flags.go:64] FLAG: --event-burst="100" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035759 2582 flags.go:64] FLAG: --event-qps="50" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035762 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035766 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035768 2582 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035773 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035776 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035779 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035782 2582 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035786 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035789 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:37:08.040773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035791 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035794 2582 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035798 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035801 2582 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035803 2582 flags.go:64] FLAG: --feature-gates="" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035807 2582 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035810 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035814 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035818 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035821 2582 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035824 2582 flags.go:64] FLAG: --help="false" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035827 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035831 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035834 2582 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035838 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035841 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035844 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035847 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035850 2582 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035854 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035857 2582 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035860 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035863 2582 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035866 2582 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:37:08.041418 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035869 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035872 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035875 2582 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035878 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035881 2582 flags.go:64] FLAG: --lock-file="" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035884 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035887 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035890 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035895 2582 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035899 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035902 2582 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035904 2582 flags.go:64] FLAG: --logging-format="text" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035907 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035910 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035914 2582 flags.go:64] FLAG: --manifest-url="" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035916 2582 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035922 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035925 2582 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035929 2582 flags.go:64] FLAG: --max-pods="110" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035932 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035935 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035938 2582 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035941 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035944 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035947 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:37:08.042103 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035950 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035957 2582 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035962 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035965 2582 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035968 2582 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035971 2582 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035974 2582 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035977 2582 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035980 2582 flags.go:64] FLAG: --port="10250" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035984 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035987 2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0137315b5138f4b7d" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035990 2582 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035993 2582 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035996 2582 flags.go:64] FLAG: --register-node="true" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.035999 2582 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036003 2582 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036005 2582 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036008 2582 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036011 2582 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036015 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036017 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036020 2582 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036023 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036027 2582 flags.go:64] FLAG: --runonce="false" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036029 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036033 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036036 2582 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:37:08.042715 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036039 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036041 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036045 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036048 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036051 2582 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036054 2582 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036057 2582 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036061 2582 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036064 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036068 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036070 2582 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036073 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036078 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036081 2582 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036084 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036088 2582 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036091 2582 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036094 2582 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036097 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036100 2582 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036103 2582 flags.go:64] FLAG: --v="2" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036107 2582 flags.go:64] FLAG: --version="false" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036111 2582 flags.go:64] FLAG: --vmodule="" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036115 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036118 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:37:08.043345 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036259 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036262 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036265 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036268 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036271 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036289 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036292 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036296 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036298 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036302 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036308 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036311 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036314 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036317 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036321 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036323 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036326 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036329 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036332 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.043974 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036335 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036337 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036340 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036343 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036345 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036348 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036351 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036353 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036357 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036359 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036362 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036365 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036367 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036370 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036373 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036376 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036378 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036381 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036384 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036387 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.044412 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036389 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036392 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036394 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036397 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036400 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036403 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036406 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036409 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036412 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036416 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036420 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036423 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036426 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036430 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036432 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036435 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036437 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036440 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036442 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036445 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.044893 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036448 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036450 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036453 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036455 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036458 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036460 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036463 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036466 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036468 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036471 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036474 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036476 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036479 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036481 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036484 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036486 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036489 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036492 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036495 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.045353 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036499 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036501 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036504 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036506 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036509 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036512 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036514 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036517 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036519 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036522 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036524 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036527 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036530 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036532 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036536 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036539 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036541 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036544 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036546 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036549 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.045852 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036551 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036554 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036556 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036559 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036562 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036564 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036567 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036569 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.036595 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036629 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036633 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036637 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036640 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036644 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036647 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036650 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.046328 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036652 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036655 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036658 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036660 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036663 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036665 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036668 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036671 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036673 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036676 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036680 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036683 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036685 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036688 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036691 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036693 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036707 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036710 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036713 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036716 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.046726 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036718 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036721 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036723 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036726 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036729 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036731 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036734 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036737 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036743 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036747 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036750 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036752 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036755 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036758 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036761 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036764 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036766 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036769 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036771 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036774 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.047200 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036776 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036779 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036783 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036785 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036788 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036790 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036793 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036796 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036798 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036801 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036803 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036806 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036808 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036811 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036814 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036816 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036819 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036821 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036824 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.047680 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036828 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036832 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036836 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036839 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036842 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036845 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036847 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036850 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036852 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036855 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036858 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036860 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036863 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036865 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036868 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036872 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036875 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036878 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036881 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.048142 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036883 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036886 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036889 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036891 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036894 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036896 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036899 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036902 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036904 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036907 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036910 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036912 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036915 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036918 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036920 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036924 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036927 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036930 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036932 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.048588 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036935 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.036939 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.043396 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.043415 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.043430 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.043435 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043491 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043496 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043500 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043503 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043505 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043508 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043511 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043514 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043516 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043519 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043522 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043524 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043527 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.049067 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043530 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043533 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043535 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043538 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043541 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043544 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043546 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043550 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043552 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043555 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043558 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043561 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043564 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043566 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043569 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043572 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043575 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043578 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043581 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043584 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.049515 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043586 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043589 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043592 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043594 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043597 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043599 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043602 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043605 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043607 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043610 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043613 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043615 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043618 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043620 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043623 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043626 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043628 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043633 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043636 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.050051 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043639 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043642 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043645 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043648 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043651 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043654 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043656 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043659 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043661 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043664 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043668 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043670 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043673 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043676 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043678 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043681 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043683 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043686 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043689 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.050495 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043691 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043694 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043713 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043716 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043719 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043721 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043724 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043727 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043730 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043733 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043736 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043738 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043741 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043744 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043747 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043750 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043753 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043755 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043758 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043761 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.050966 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043763 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043766 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043768 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043771 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043774 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043777 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043779 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043782 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043785 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043789 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043793 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043796 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043798 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043801 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043804 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.051456 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.043833 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043978 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043982 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043986 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043989 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043991 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043994 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043997 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.043999 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044002 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044005 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044007 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044010 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044013 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044016 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044018 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044021 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044023 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044026 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044028 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044031 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.051869 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044034 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044037 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044040 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044042 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044045 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044047 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044050 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044052 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044055 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044057 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044060 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044062 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044065 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044067 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044070 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044072 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044075 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044078 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044080 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.052657 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044083 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044086 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044089 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044091 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044094 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044096 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044099 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044102 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044104 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044108 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044111 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044114 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044117 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044119 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044122 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044125 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044128 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044130 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044133 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.053189 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044135 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044138 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044140 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044143 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044145 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044148 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044150 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044153 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044155 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044158 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044161 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044164 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044166 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044169 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044171 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044174 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044176 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044181 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044184 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.053639 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044187 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044191 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044193 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044196 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044199 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044201 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044204 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044207 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044209 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044212 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044215 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044217 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044220 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044222 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044225 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044228 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044230 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044233 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044235 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.054098 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044238 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044240 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044243 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044245 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044252 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044255 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044257 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044260 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044263 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044265 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.044288 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044321 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044326 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044329 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044332 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:08.054535 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044335 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044337 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044340 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044343 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044361 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044365 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044369 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044372 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044375 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044378 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044382 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044385 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044387 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044390 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044393 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044395 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044398 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044402 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044406 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:08.054935 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044408 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044411 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044414 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044418 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044420 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044423 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044426 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044428 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044431 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044434 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044436 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044439 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044442 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044444 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044447 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044449 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044452 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044454 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044457 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:08.055385 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044460 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044462 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044465 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044467 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044470 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044473 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044475 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044478 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044481 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044483 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044486 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044489 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044492 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044494 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044497 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044499 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044503 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044506 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044509 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044511 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:08.055886 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044514 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044517 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044519 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044522 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044525 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044528 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044530 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044533 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044536 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044538 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044541 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044544 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044546 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044549 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044551 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044554 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044556 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044559 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044562 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044565 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:08.056366 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044567 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044570 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044572 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044575 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044578 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044581 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044584 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044586 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044590 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044593 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044596 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044598 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044601 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044604 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044606 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044609 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044612 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044615 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044619 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:08.057145 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044622 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044625 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044628 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044631 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:08.044634 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.044723 2582 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.046893 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.047971 2582 server.go:1014] "Starting client certificate rotation" Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.048071 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.048107 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.050131 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:37:08.057594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.057354 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:37:08.063352 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.063331 2582 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:37:08.066724 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.066692 2582 log.go:25] "Validated CRI v1 image API" Nov 03 18:37:08.067565 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.067543 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:37:08.072847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.072828 2582 fs.go:135] Filesystem UUIDs: map[13337f76-2d5d-414e-acff-5855b802586d:/dev/nvme0n1p4 6e91736e-d413-4447-96e6-0eb14e69c40a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 18:37:08.072899 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.072847 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 18:37:08.077961 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.077838 2582 manager.go:217] Machine: {Timestamp:2026-11-03 18:37:08.076430067 +0000 UTC m=+0.331418609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099965 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec253c03869f63ed55da7a07b82e13c9 SystemUUID:ec253c03-869f-63ed-55da-7a07b82e13c9 BootID:4e52d022-1f13-48cc-86e9-ac5a819e27f3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c4:de:0a:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c4:de:0a:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:85:1f:fe:64:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:37:08.078057 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.077965 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:37:08.078086 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.078073 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:37:08.081291 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.081265 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:37:08.081420 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.081294 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-95.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:37:08.081466 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.081423 2582 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:37:08.081466 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.081431 2582 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:37:08.081516 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.081466 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:37:08.081516 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.081484 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:37:08.082499 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.082490 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:37:08.085457 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.085445 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:37:08.085576 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.085567 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:37:08.086399 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.086390 2582 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:37:08.086432 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.086404 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:37:08.086432 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.086415 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:37:08.086432 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.086424 2582 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:37:08.086432 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.086432 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:37:08.086794 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.086781 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:37:08.088910 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.088892 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:37:08.093550 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.093535 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:37:08.094553 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094540 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:37:08.094599 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094572 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:37:08.094773 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094762 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:37:08.094811 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094779 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:37:08.094811 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094786 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:37:08.094811 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094798 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:37:08.094811 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094807 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:37:08.094915 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.094814 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:37:08.095311 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.095303 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:37:08.095338 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.095312 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:37:08.095338 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.095320 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:37:08.095338 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.095326 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:37:08.095421 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.095340 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:37:08.095421 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.095356 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:37:08.096949 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.096938 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:37:08.096985 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.096952 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:37:08.100771 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.100756 2582 server.go:1263] "Started kubelet" Nov 03 18:37:08.100901 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.100848 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:37:08.100945 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.100860 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:37:08.100973 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.100946 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:37:08.101549 ip-10-0-128-95 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:37:08.107037 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.105413 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:37:08.107037 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.106412 2582 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:37:08.110477 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.110460 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:37:08.110608 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.110569 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 18:37:08.110967 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.110952 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:37:08.111105 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.111084 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" Nov 03 18:37:08.111847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.111675 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:37:08.111847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.111674 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:37:08.111847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.111710 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:37:08.111847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.111767 2582 factory.go:55] Registering systemd factory Nov 03 18:37:08.111847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.111797 2582 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:37:08.111847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.111807 2582 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:37:08.111847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.111808 2582 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:37:08.112096 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.112023 2582 factory.go:153] Registering CRI-O factory Nov 03 18:37:08.112096 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.112037 2582 factory.go:223] Registration of the crio container factory successfully Nov 03 18:37:08.112096 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.112076 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:37:08.112096 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.112090 2582 factory.go:103] Registering Raw factory Nov 03 18:37:08.112240 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.112101 2582 manager.go:1199] Started watching for new ooms in manager Nov 03 18:37:08.112949 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.112938 2582 manager.go:319] Starting recovery of all containers Nov 03 18:37:08.124444 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.124427 2582 manager.go:324] Recovery completed Nov 03 18:37:08.130467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.130452 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.132487 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.132468 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.132558 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.132500 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.132558 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.132528 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.133126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.133112 2582 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:37:08.133126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.133124 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:37:08.133252 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.133143 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:37:08.133252 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.133165 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.134013 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.133995 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.134072 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.134028 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.134072 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.134041 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.134459 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.134449 2582 policy_none.go:50] "Start" Nov 03 18:37:08.134489 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.134472 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:37:08.134489 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.134482 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:37:08.135560 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.135549 2582 policy_none.go:44] "Start" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.172299 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.175276 2582 manager.go:341] "Starting Device Plugin manager" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.175310 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.175321 2582 server.go:87] "Starting device plugin registration server" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.175628 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.175643 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.175889 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.175974 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.175982 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.176568 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.177008 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" Nov 03 18:37:08.182662 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.181870 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:37:08.183237 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.183154 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:37:08.183237 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.183174 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:37:08.183237 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.183193 2582 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:37:08.183237 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.183225 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 18:37:08.252354 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.252322 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-95.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 18:37:08.252354 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.252330 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-95.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 18:37:08.253375 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.252327 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-95.us-west-2.compute.internal.18e49cddd08a445f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-95.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-95.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:08.100727903 +0000 UTC m=+0.355716445,LastTimestamp:2026-11-03 18:37:08.100727903 +0000 UTC m=+0.355716445,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-95.us-west-2.compute.internal,}" Nov 03 18:37:08.277968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.277894 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.278877 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.278860 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.278939 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.278897 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.278939 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.278911 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.278939 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.278938 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.283569 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.283551 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal"] Nov 03 18:37:08.283645 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.283602 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.284405 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.284387 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.284478 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.284417 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.284478 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.284429 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.286669 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.286658 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.286859 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.286823 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.286859 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.286856 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.287533 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.287516 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.287621 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.287543 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.287621 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.287512 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.287621 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.287557 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.287621 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.287590 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.287621 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.287606 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.288847 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.288833 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.288894 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.288857 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.289635 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.289616 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.289731 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.289648 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.289731 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.289662 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.313288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.313261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.313379 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.313290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31283f871e3314b8eda74a4d6e8f72-config\") pod \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"3e31283f871e3314b8eda74a4d6e8f72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.313379 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.313319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.315752 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.315732 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.319101 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.319085 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.332352 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.332320 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 18:37:08.332448 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.332312 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-95.us-west-2.compute.internal.18e49cddd26ee113 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-95.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-95.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-95.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:08.132487443 +0000 UTC m=+0.387475988,LastTimestamp:2026-11-03 18:37:08.132487443 +0000 UTC m=+0.387475988,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-95.us-west-2.compute.internal,}" Nov 03 18:37:08.337428 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.337412 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:37:08.337496 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.337470 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mx2wb" Nov 03 18:37:08.413890 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.413861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.413890 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.413889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.414077 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.413909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31283f871e3314b8eda74a4d6e8f72-config\") pod \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"3e31283f871e3314b8eda74a4d6e8f72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.414077 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.413965 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31283f871e3314b8eda74a4d6e8f72-config\") pod \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"3e31283f871e3314b8eda74a4d6e8f72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.414077 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.413966 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.414077 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.413966 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.421008 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.420989 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:08.421106 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.421012 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:08.421189 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.421175 2582 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-128-95.us-west-2.compute.internal\" not found" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.422152 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.422138 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:08.561687 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:08.561613 2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-95.us-west-2.compute.internal\" not found" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.619104 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.619070 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.621307 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.621290 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:08.622280 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.622260 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:08.622372 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.622297 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:08.622372 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.622309 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:08.622372 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.622342 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.623521 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.623505 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.681178 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.681151 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.711449 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.711426 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.798524 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.798495 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:08.799406 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.799393 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:08.897792 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:08.897723 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:09.048472 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.048446 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 18:37:09.049023 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.048567 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:37:09.049023 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.048575 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:37:09.049023 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.048622 2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://acc3b4ca8eba54ce98364924c8229c13-670cf6387112f978.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-95.us-west-2.compute.internal.18e49cddd26ee113\": read tcp 10.0.128.95:40592->18.218.183.169:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-95.us-west-2.compute.internal.18e49cddd26ee113 default 4816 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-95.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-95.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-95.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:08 +0000 UTC,LastTimestamp:2026-11-03 18:37:08.284403671 +0000 UTC m=+0.539392217,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-95.us-west-2.compute.internal,}" Nov 03 18:37:09.087337 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.087307 2582 apiserver.go:52] "Watching apiserver" Nov 03 18:37:09.110989 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.110961 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 18:37:09.341346 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.341252 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:32:08 +0000 UTC" deadline="2028-07-02 01:05:47.723346525 +0000 UTC" Nov 03 18:37:09.341346 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.341287 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14550h28m38.382062681s" Nov 03 18:37:09.388858 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.388830 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:37:09.389175 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.389154 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-nl6fg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h","openshift-cluster-node-tuning-operator/tuned-clxql","openshift-image-registry/node-ca-hmqxx","openshift-multus/multus-64c5w","openshift-multus/multus-additional-cni-plugins-lv47h","openshift-multus/network-metrics-daemon-l5g7x","openshift-network-diagnostics/network-check-target-6bn8h","kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal","openshift-dns/node-resolver-tlhcd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-qdczr","openshift-ovn-kubernetes/ovnkube-node-t92q5","kube-system/global-pull-secret-syncer-t4dcp"] Nov 03 18:37:09.390965 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.390943 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:09.391059 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.391038 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:09.392376 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.392360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.393615 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.393596 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.394785 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.394755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.395793 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.395775 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.397111 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.397093 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.398304 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.398290 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.399441 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.399423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:09.399520 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.399499 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:09.400654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.400642 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.402096 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.402081 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.403438 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.403420 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.404595 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.404577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.404666 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.404635 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:09.413565 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.413545 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:37:09.420076 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-socket-dir-parent\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.420167 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.420167 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3441fcb6-fc61-446d-a986-79d01440c01b-host\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.420167 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-tmp\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.420167 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420136 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.420392 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.420392 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-kubelet-config\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.420392 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-dbus\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.420392 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:09.420392 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjs9k\" (UniqueName: \"kubernetes.io/projected/a59c6f41-01de-40f1-9725-1ed9bd6784d6-kube-api-access-gjs9k\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.420392 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-host\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hdv\" (UniqueName: \"kubernetes.io/projected/175d5971-984f-4e61-bdaa-f1b34609f0e4-kube-api-access-h8hdv\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420468 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysconfig\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-kubernetes\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-systemd\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cnibin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-kubelet\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3441fcb6-fc61-446d-a986-79d01440c01b-serviceca\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420604 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-node-log\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-config\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-kubelet\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee858f-ae52-4de9-b576-ed63754072f9-konnectivity-ca\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-var-lib-kubelet\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-k8s-cni-cncf-io\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.420750 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjg9j\" (UniqueName: \"kubernetes.io/projected/c3e13ce6-21d8-4d9c-b589-46699c28b856-kube-api-access-gjg9j\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420800 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-socket-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-registration-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420850 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-bin\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-cnibin\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420892 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8g8\" (UniqueName: \"kubernetes.io/projected/3441fcb6-fc61-446d-a986-79d01440c01b-kube-api-access-7c8g8\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420909 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovn-node-metrics-cert\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-system-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420965 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-modprobe-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.420989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421036 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-sys-fs\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-netns\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-ovn\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-run\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.421467 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cni-binary-copy\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-device-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-systemd-units\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-etc-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6319a2b4-76e5-42dc-8097-a52570ed7f2c-hosts-file\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-daemon-config\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-conf\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-multus\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-sys\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-etc-kubernetes\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffx2\" (UniqueName: \"kubernetes.io/projected/1d027ae4-4c9c-4beb-8134-3995b81d32c1-kube-api-access-mffx2\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-slash\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.422203 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59c6f41-01de-40f1-9725-1ed9bd6784d6-host-slash\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-conf-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421604 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-log-socket\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6319a2b4-76e5-42dc-8097-a52570ed7f2c-tmp-dir\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-lib-modules\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-bin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421734 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-multus-certs\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-var-lib-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421806 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6fd\" (UniqueName: \"kubernetes.io/projected/6319a2b4-76e5-42dc-8097-a52570ed7f2c-kube-api-access-2k6fd\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59c6f41-01de-40f1-9725-1ed9bd6784d6-iptables-alerter-script\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-binary-copy\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421889 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-netns\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-hostroot\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.421984 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlpc\" (UniqueName: \"kubernetes.io/projected/d4c6be41-6db5-4be6-8cac-76b00171f6af-kube-api-access-2qlpc\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.422706 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-tuned\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-netd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7lf\" (UniqueName: \"kubernetes.io/projected/af92784a-e931-457d-a327-5fbba2f37af5-kube-api-access-sp7lf\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422098 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-systemd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-script-lib\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422175 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0ee858f-ae52-4de9-b576-ed63754072f9-agent-certs\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-system-cni-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-env-overrides\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.423200 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.422284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrnd\" (UniqueName: \"kubernetes.io/projected/ce2ce282-8a19-40f3-a97f-a270daedbe66-kube-api-access-pmrnd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.450551 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.450521 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:37:09.450551 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.450522 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:37:09.450735 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.450560 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r88kn\"" Nov 03 18:37:09.451019 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.450996 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:09.451097 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.451065 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:09.451158 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.451121 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d4mhq\"" Nov 03 18:37:09.452682 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.452667 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:37:09.452682 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.452675 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:37:09.452763 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.452667 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczj\"" Nov 03 18:37:09.453072 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.453053 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:37:09.453148 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.453125 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:37:09.453410 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.453398 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-btttq\"" Nov 03 18:37:09.453573 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.453560 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:37:09.453805 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.453793 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:37:09.454532 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.454516 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:37:09.454592 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.454533 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:37:09.454626 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.454604 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lzn26\"" Nov 03 18:37:09.455867 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.455848 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:37:09.455966 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.455868 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:37:09.455966 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.455908 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:37:09.455966 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.455950 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fft52\"" Nov 03 18:37:09.457677 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.457659 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:37:09.501666 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.501645 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cxqpk\"" Nov 03 18:37:09.502116 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.502091 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:37:09.503885 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.503861 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:37:09.503990 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.503884 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:37:09.503990 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.503912 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zsb8x\"" Nov 03 18:37:09.503990 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.503946 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxjph\"" Nov 03 18:37:09.503990 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.503971 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:09.506366 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506352 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:09.506425 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506405 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:37:09.506478 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506431 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:37:09.506542 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506504 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:37:09.506542 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506510 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:37:09.506542 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506536 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:37:09.506704 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506569 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:37:09.506704 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506584 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:37:09.506704 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.506596 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8frn6" Nov 03 18:37:09.523220 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-netns\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523300 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-ovn\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523300 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-run\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.523300 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cni-binary-copy\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.523391 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523299 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-netns\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523391 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523302 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-ovn\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523391 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-device-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.523391 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523330 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-run\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.523391 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523347 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-device-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.523391 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-systemd-units\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523391 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523382 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-etc-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6319a2b4-76e5-42dc-8097-a52570ed7f2c-hosts-file\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523464 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-systemd-units\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523468 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-daemon-config\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523521 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6319a2b4-76e5-42dc-8097-a52570ed7f2c-hosts-file\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523523 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-etc-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523535 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-conf\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-multus\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.523636 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-sys\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-etc-kubernetes\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffx2\" (UniqueName: \"kubernetes.io/projected/1d027ae4-4c9c-4beb-8134-3995b81d32c1-kube-api-access-mffx2\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523712 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-sys\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523729 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-multus\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523738 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-slash\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523757 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-etc-kubernetes\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523783 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-slash\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523782 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59c6f41-01de-40f1-9725-1ed9bd6784d6-host-slash\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523821 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-conf-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.523873 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523882 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cni-binary-copy\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523891 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-conf\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-log-socket\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523914 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-log-socket\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.523938 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret podName:8da56f0f-6ab9-4eb3-ad1d-35ce98845ada nodeName:}" failed. No retries permitted until 2026-11-03 18:37:10.023919263 +0000 UTC m=+2.278907818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret") pod "global-pull-secret-syncer-t4dcp" (UID: "8da56f0f-6ab9-4eb3-ad1d-35ce98845ada") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:09.524334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523943 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59c6f41-01de-40f1-9725-1ed9bd6784d6-host-slash\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6319a2b4-76e5-42dc-8097-a52570ed7f2c-tmp-dir\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-lib-modules\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-bin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524025 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-multus-certs\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-var-lib-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524085 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-daemon-config\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6fd\" (UniqueName: \"kubernetes.io/projected/6319a2b4-76e5-42dc-8097-a52570ed7f2c-kube-api-access-2k6fd\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.523962 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-conf-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524118 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59c6f41-01de-40f1-9725-1ed9bd6784d6-iptables-alerter-script\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524138 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524158 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-bin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524168 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-multus-certs\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524186 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-var-lib-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-binary-copy\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524219 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-lib-modules\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.525221 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524259 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6319a2b4-76e5-42dc-8097-a52570ed7f2c-tmp-dir\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-netns\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-hostroot\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlpc\" (UniqueName: \"kubernetes.io/projected/d4c6be41-6db5-4be6-8cac-76b00171f6af-kube-api-access-2qlpc\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524349 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-netns\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524366 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-tuned\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-netd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7lf\" (UniqueName: \"kubernetes.io/projected/af92784a-e931-457d-a327-5fbba2f37af5-kube-api-access-sp7lf\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-systemd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-script-lib\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524503 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0ee858f-ae52-4de9-b576-ed63754072f9-agent-certs\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524529 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-system-cni-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-env-overrides\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrnd\" (UniqueName: \"kubernetes.io/projected/ce2ce282-8a19-40f3-a97f-a270daedbe66-kube-api-access-pmrnd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-socket-dir-parent\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3441fcb6-fc61-446d-a986-79d01440c01b-host\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.526126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-tmp\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524476 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-netd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524715 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-kubelet-config\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524820 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-binary-copy\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524834 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-dbus\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524857 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59c6f41-01de-40f1-9725-1ed9bd6784d6-iptables-alerter-script\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524880 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-systemd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjs9k\" (UniqueName: \"kubernetes.io/projected/a59c6f41-01de-40f1-9725-1ed9bd6784d6-kube-api-access-gjs9k\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524860 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-hostroot\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524932 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-socket-dir-parent\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-host\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524932 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-kubelet-config\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525009 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-system-cni-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.527021 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525054 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525004 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-dbus\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525192 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-host\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524969 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3441fcb6-fc61-446d-a986-79d01440c01b-host\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hdv\" (UniqueName: \"kubernetes.io/projected/175d5971-984f-4e61-bdaa-f1b34609f0e4-kube-api-access-h8hdv\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524822 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.524827 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525235 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-script-lib\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysconfig\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525369 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-env-overrides\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-kubernetes\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525469 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysconfig\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-systemd\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cnibin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525551 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-systemd\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-kubelet\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3441fcb6-fc61-446d-a986-79d01440c01b-serviceca\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.527743 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525599 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cnibin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525610 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-kubernetes\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-node-log\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525653 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-node-log\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525656 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-config\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-kubelet\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee858f-ae52-4de9-b576-ed63754072f9-konnectivity-ca\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-var-lib-kubelet\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525868 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-k8s-cni-cncf-io\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjg9j\" (UniqueName: \"kubernetes.io/projected/c3e13ce6-21d8-4d9c-b589-46699c28b856-kube-api-access-gjg9j\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525934 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-socket-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-registration-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-bin\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.525997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-cnibin\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8g8\" (UniqueName: \"kubernetes.io/projected/3441fcb6-fc61-446d-a986-79d01440c01b-kube-api-access-7c8g8\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovn-node-metrics-cert\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528325 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-system-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-modprobe-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526135 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-config\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-sys-fs\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526186 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3441fcb6-fc61-446d-a986-79d01440c01b-serviceca\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526207 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-kubelet\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526251 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-kubelet\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526313 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-sys-fs\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526351 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-bin\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526415 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-var-lib-kubelet\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526489 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-k8s-cni-cncf-io\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.526571 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.526631 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:10.026601084 +0000 UTC m=+2.281589619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526642 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-system-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526658 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee858f-ae52-4de9-b576-ed63754072f9-konnectivity-ca\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.528903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526692 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-cnibin\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526765 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-modprobe-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526861 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.526909 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.527008 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-registration-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.527022 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-socket-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.529402 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-tuned\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.529501 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-tmp\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.529723 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.529563 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0ee858f-ae52-4de9-b576-ed63754072f9-agent-certs\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.530126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.529773 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovn-node-metrics-cert\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.561104 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.561070 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:09.589988 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.589957 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:09.589988 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.589983 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:09.590182 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.590009 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m5fj8 for pod openshift-network-diagnostics/network-check-target-6bn8h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:09.590182 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:09.590092 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8 podName:6b7d3c74-e08e-44b1-ab35-ced1363278f0 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:10.090056846 +0000 UTC m=+2.345045380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m5fj8" (UniqueName: "kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8") pod "network-check-target-6bn8h" (UID: "6b7d3c74-e08e-44b1-ab35-ced1363278f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:09.592142 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.592073 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6fd\" (UniqueName: \"kubernetes.io/projected/6319a2b4-76e5-42dc-8097-a52570ed7f2c-kube-api-access-2k6fd\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.593205 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.593178 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffx2\" (UniqueName: \"kubernetes.io/projected/1d027ae4-4c9c-4beb-8134-3995b81d32c1-kube-api-access-mffx2\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.593291 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.593225 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7lf\" (UniqueName: \"kubernetes.io/projected/af92784a-e931-457d-a327-5fbba2f37af5-kube-api-access-sp7lf\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.593338 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.593287 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hdv\" (UniqueName: \"kubernetes.io/projected/175d5971-984f-4e61-bdaa-f1b34609f0e4-kube-api-access-h8hdv\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.593338 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.593308 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrnd\" (UniqueName: \"kubernetes.io/projected/ce2ce282-8a19-40f3-a97f-a270daedbe66-kube-api-access-pmrnd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.593536 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.593513 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjs9k\" (UniqueName: \"kubernetes.io/projected/a59c6f41-01de-40f1-9725-1ed9bd6784d6-kube-api-access-gjs9k\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.596488 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.596469 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlpc\" (UniqueName: \"kubernetes.io/projected/d4c6be41-6db5-4be6-8cac-76b00171f6af-kube-api-access-2qlpc\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.598204 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.598178 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8g8\" (UniqueName: \"kubernetes.io/projected/3441fcb6-fc61-446d-a986-79d01440c01b-kube-api-access-7c8g8\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.598416 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.598400 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjg9j\" (UniqueName: \"kubernetes.io/projected/c3e13ce6-21d8-4d9c-b589-46699c28b856-kube-api-access-gjg9j\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:09.726393 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.726355 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:09.744769 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.744737 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:09.749754 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.749728 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:09.773893 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.773862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-64c5w" Nov 03 18:37:09.784865 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.784844 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:09.793832 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.793812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:09.799978 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:09.799952 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c6be41_6db5_4be6_8cac_76b00171f6af.slice/crio-616adb26588f4e73d754a9769c2169cdda6eb4671fe134cc8d42066bb18659f9 WatchSource:0}: Error finding container 616adb26588f4e73d754a9769c2169cdda6eb4671fe134cc8d42066bb18659f9: Status 404 returned error can't find the container with id 616adb26588f4e73d754a9769c2169cdda6eb4671fe134cc8d42066bb18659f9 Nov 03 18:37:09.800653 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:09.800629 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a249de0ac9079d1551a521e9f3cd5d3.slice/crio-2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8 WatchSource:0}: Error finding container 2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8: Status 404 returned error can't find the container with id 2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8 Nov 03 18:37:09.806869 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.806849 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:37:09.807892 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:09.807865 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3441fcb6_fc61_446d_a986_79d01440c01b.slice/crio-24d36e323082a4a69a27793e13bb654eb1bb38322f97e064b4612f5549f7a05b WatchSource:0}: Error finding container 24d36e323082a4a69a27793e13bb654eb1bb38322f97e064b4612f5549f7a05b: Status 404 returned error can't find the container with id 24d36e323082a4a69a27793e13bb654eb1bb38322f97e064b4612f5549f7a05b Nov 03 18:37:09.808124 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:09.808101 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e31283f871e3314b8eda74a4d6e8f72.slice/crio-bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e WatchSource:0}: Error finding container bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e: Status 404 returned error can't find the container with id bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e Nov 03 18:37:09.813887 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.813868 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:09.821102 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:09.821073 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf92784a_e931_457d_a327_5fbba2f37af5.slice/crio-c6dc24380b0a21f21f405e06645803c376cc57765ed14707f317d94c2b5f0430 WatchSource:0}: Error finding container c6dc24380b0a21f21f405e06645803c376cc57765ed14707f317d94c2b5f0430: Status 404 returned error can't find the container with id c6dc24380b0a21f21f405e06645803c376cc57765ed14707f317d94c2b5f0430 Nov 03 18:37:09.832447 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.832421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:09.841033 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:09.841006 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59c6f41_01de_40f1_9725_1ed9bd6784d6.slice/crio-f028eb3cb7a5ff0b6e9951ad48e5cd1d78dad1da208c47a5e7055b8836bf70cb WatchSource:0}: Error finding container f028eb3cb7a5ff0b6e9951ad48e5cd1d78dad1da208c47a5e7055b8836bf70cb: Status 404 returned error can't find the container with id f028eb3cb7a5ff0b6e9951ad48e5cd1d78dad1da208c47a5e7055b8836bf70cb Nov 03 18:37:09.849300 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:09.849281 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:09.856889 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:09.856858 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2ce282_8a19_40f3_a97f_a270daedbe66.slice/crio-a2acd7e013068985190ce35177b09ebb3a129c83c9755935aaf7ea05b2fb40af WatchSource:0}: Error finding container a2acd7e013068985190ce35177b09ebb3a129c83c9755935aaf7ea05b2fb40af: Status 404 returned error can't find the container with id a2acd7e013068985190ce35177b09ebb3a129c83c9755935aaf7ea05b2fb40af Nov 03 18:37:10.030092 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.030039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:10.030302 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.030174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:10.030302 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.030227 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:10.030302 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.030287 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:10.030458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.030316 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret podName:8da56f0f-6ab9-4eb3-ad1d-35ce98845ada nodeName:}" failed. No retries permitted until 2026-11-03 18:37:11.030297063 +0000 UTC m=+3.285285615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret") pod "global-pull-secret-syncer-t4dcp" (UID: "8da56f0f-6ab9-4eb3-ad1d-35ce98845ada") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:10.030458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.030344 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:11.030325773 +0000 UTC m=+3.285314322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:10.130943 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.130844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:10.131453 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.131011 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:10.131453 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.131034 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:10.131453 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.131045 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m5fj8 for pod openshift-network-diagnostics/network-check-target-6bn8h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:10.131453 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:10.131097 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8 podName:6b7d3c74-e08e-44b1-ab35-ced1363278f0 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:11.131083199 +0000 UTC m=+3.386071730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5fj8" (UniqueName: "kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8") pod "network-check-target-6bn8h" (UID: "6b7d3c74-e08e-44b1-ab35-ced1363278f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:10.186208 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.186176 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:10.187842 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.187798 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qdczr" event={"ID":"a59c6f41-01de-40f1-9725-1ed9bd6784d6","Type":"ContainerStarted","Data":"f028eb3cb7a5ff0b6e9951ad48e5cd1d78dad1da208c47a5e7055b8836bf70cb"} Nov 03 18:37:10.188967 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.188939 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" event={"ID":"af92784a-e931-457d-a327-5fbba2f37af5","Type":"ContainerStarted","Data":"c6dc24380b0a21f21f405e06645803c376cc57765ed14707f317d94c2b5f0430"} Nov 03 18:37:10.190033 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.190010 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"3e31283f871e3314b8eda74a4d6e8f72","Type":"ContainerStarted","Data":"bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e"} Nov 03 18:37:10.191085 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.191065 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmqxx" event={"ID":"3441fcb6-fc61-446d-a986-79d01440c01b","Type":"ContainerStarted","Data":"24d36e323082a4a69a27793e13bb654eb1bb38322f97e064b4612f5549f7a05b"} Nov 03 18:37:10.192159 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.192139 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerStarted","Data":"616adb26588f4e73d754a9769c2169cdda6eb4671fe134cc8d42066bb18659f9"} Nov 03 18:37:10.193234 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.193215 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"8a249de0ac9079d1551a521e9f3cd5d3","Type":"ContainerStarted","Data":"2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8"} Nov 03 18:37:10.194685 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.194665 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"a2acd7e013068985190ce35177b09ebb3a129c83c9755935aaf7ea05b2fb40af"} Nov 03 18:37:10.475817 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.475724 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:10.509594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.509541 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:09 +0000 UTC" deadline="2028-05-02 09:56:07.607097093 +0000 UTC" Nov 03 18:37:10.509594 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:10.509585 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13095h18m57.097513863s" Nov 03 18:37:10.596510 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:10.596465 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6319a2b4_76e5_42dc_8097_a52570ed7f2c.slice/crio-a3d9cde95c420b57810027daa9e2671d5d8c627f3d08c8ef86afbd9ddde693a2 WatchSource:0}: Error finding container a3d9cde95c420b57810027daa9e2671d5d8c627f3d08c8ef86afbd9ddde693a2: Status 404 returned error can't find the container with id a3d9cde95c420b57810027daa9e2671d5d8c627f3d08c8ef86afbd9ddde693a2 Nov 03 18:37:10.646160 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:10.646131 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175d5971_984f_4e61_bdaa_f1b34609f0e4.slice/crio-10198484f7faa1acdd12bc9c0de60f30e9b312df76f8cc8e2df5a25f21f96bb6 WatchSource:0}: Error finding container 10198484f7faa1acdd12bc9c0de60f30e9b312df76f8cc8e2df5a25f21f96bb6: Status 404 returned error can't find the container with id 10198484f7faa1acdd12bc9c0de60f30e9b312df76f8cc8e2df5a25f21f96bb6 Nov 03 18:37:10.676804 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:10.676758 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ee858f_ae52_4de9_b576_ed63754072f9.slice/crio-6a3f01e433aeb94fed773b3d809c8a8c891d0f710f37cd1cc599433ce9e614b1 WatchSource:0}: Error finding container 6a3f01e433aeb94fed773b3d809c8a8c891d0f710f37cd1cc599433ce9e614b1: Status 404 returned error can't find the container with id 6a3f01e433aeb94fed773b3d809c8a8c891d0f710f37cd1cc599433ce9e614b1 Nov 03 18:37:10.712343 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:10.712313 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d027ae4_4c9c_4beb_8134_3995b81d32c1.slice/crio-6c368b178ed03df9d3eb3f49c050a2c621500cfab5b82170e6e433f7d2ceca8c WatchSource:0}: Error finding container 6c368b178ed03df9d3eb3f49c050a2c621500cfab5b82170e6e433f7d2ceca8c: Status 404 returned error can't find the container with id 6c368b178ed03df9d3eb3f49c050a2c621500cfab5b82170e6e433f7d2ceca8c Nov 03 18:37:11.038250 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.038211 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:11.038453 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.038309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:11.038534 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.038460 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:11.038534 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.038527 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:13.038508485 +0000 UTC m=+5.293497042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:11.038653 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.038596 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:11.038653 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.038631 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret podName:8da56f0f-6ab9-4eb3-ad1d-35ce98845ada nodeName:}" failed. No retries permitted until 2026-11-03 18:37:13.038619678 +0000 UTC m=+5.293608215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret") pod "global-pull-secret-syncer-t4dcp" (UID: "8da56f0f-6ab9-4eb3-ad1d-35ce98845ada") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:11.138936 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.138899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:11.139535 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.139111 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:11.139535 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.139134 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:11.139535 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.139148 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m5fj8 for pod openshift-network-diagnostics/network-check-target-6bn8h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:11.139535 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.139205 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8 podName:6b7d3c74-e08e-44b1-ab35-ced1363278f0 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:13.139187565 +0000 UTC m=+5.394176115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5fj8" (UniqueName: "kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8") pod "network-check-target-6bn8h" (UID: "6b7d3c74-e08e-44b1-ab35-ced1363278f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:11.183738 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.183687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:11.183920 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.183834 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:11.183987 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.183926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:11.184041 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.184004 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:11.184102 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.184066 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:11.184175 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:11.184148 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:11.209150 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.209015 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-64c5w" event={"ID":"1d027ae4-4c9c-4beb-8134-3995b81d32c1","Type":"ContainerStarted","Data":"6c368b178ed03df9d3eb3f49c050a2c621500cfab5b82170e6e433f7d2ceca8c"} Nov 03 18:37:11.214540 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.214449 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nl6fg" event={"ID":"d0ee858f-ae52-4de9-b576-ed63754072f9","Type":"ContainerStarted","Data":"6a3f01e433aeb94fed773b3d809c8a8c891d0f710f37cd1cc599433ce9e614b1"} Nov 03 18:37:11.233282 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.233214 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlhcd" event={"ID":"6319a2b4-76e5-42dc-8097-a52570ed7f2c","Type":"ContainerStarted","Data":"a3d9cde95c420b57810027daa9e2671d5d8c627f3d08c8ef86afbd9ddde693a2"} Nov 03 18:37:11.235404 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.235337 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clxql" event={"ID":"175d5971-984f-4e61-bdaa-f1b34609f0e4","Type":"ContainerStarted","Data":"10198484f7faa1acdd12bc9c0de60f30e9b312df76f8cc8e2df5a25f21f96bb6"} Nov 03 18:37:11.511323 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.511207 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:09 +0000 UTC" deadline="2028-07-08 03:41:29.547088705 +0000 UTC" Nov 03 18:37:11.511323 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:11.511286 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14697h4m18.035808714s" Nov 03 18:37:13.056403 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:13.056327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:13.056403 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:13.056392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:13.057076 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.056529 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:13.057076 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.056589 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret podName:8da56f0f-6ab9-4eb3-ad1d-35ce98845ada nodeName:}" failed. No retries permitted until 2026-11-03 18:37:17.056570983 +0000 UTC m=+9.311559548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret") pod "global-pull-secret-syncer-t4dcp" (UID: "8da56f0f-6ab9-4eb3-ad1d-35ce98845ada") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:13.057076 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.056656 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:13.057076 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.056690 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:17.056679535 +0000 UTC m=+9.311668078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:13.156775 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:13.156742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:13.156954 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.156928 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:13.156954 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.156953 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:13.157058 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.156967 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m5fj8 for pod openshift-network-diagnostics/network-check-target-6bn8h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:13.157058 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.157031 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8 podName:6b7d3c74-e08e-44b1-ab35-ced1363278f0 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:17.157012453 +0000 UTC m=+9.412000996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5fj8" (UniqueName: "kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8") pod "network-check-target-6bn8h" (UID: "6b7d3c74-e08e-44b1-ab35-ced1363278f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:13.184226 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:13.184196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:13.184367 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.184325 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:13.184421 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:13.184403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:13.184506 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.184483 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:13.184581 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:13.184543 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:13.184669 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:13.184614 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:14.273690 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:14.273654 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="c7fb616a25b28d842d00541ac59e578cced126b12fcb4ee19aef37a0623e9eef" exitCode=0 Nov 03 18:37:14.274308 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:14.273725 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerDied","Data":"c7fb616a25b28d842d00541ac59e578cced126b12fcb4ee19aef37a0623e9eef"} Nov 03 18:37:15.185016 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:15.184251 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:15.185016 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:15.184498 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:15.185016 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:15.184454 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:15.185016 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:15.184603 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:15.185016 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:15.184725 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:15.185016 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:15.184809 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:15.280236 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:15.279966 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"3e31283f871e3314b8eda74a4d6e8f72","Type":"ContainerStarted","Data":"9aaf1c360be2cfb4635430176c865f752cb5fe6b735f8f5bf852e0d7d194e0d0"} Nov 03 18:37:15.291940 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:15.291490 2582 generic.go:356] "Generic (PLEG): container finished" podID="8a249de0ac9079d1551a521e9f3cd5d3" containerID="6e559dd37ee6a38b4c4a73dbadc1b90e667a6b08e0bacc4f17163e4dc4882d59" exitCode=0 Nov 03 18:37:15.291940 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:15.291529 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"8a249de0ac9079d1551a521e9f3cd5d3","Type":"ContainerDied","Data":"6e559dd37ee6a38b4c4a73dbadc1b90e667a6b08e0bacc4f17163e4dc4882d59"} Nov 03 18:37:15.513799 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:15.513743 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" podStartSLOduration=7.51372362 podStartE2EDuration="7.51372362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:15.39702132 +0000 UTC m=+7.652009872" watchObservedRunningTime="2026-11-03 18:37:15.51372362 +0000 UTC m=+7.768712170" Nov 03 18:37:16.297324 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:16.297286 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmqxx" event={"ID":"3441fcb6-fc61-446d-a986-79d01440c01b","Type":"ContainerStarted","Data":"48c801ba8d080894fd1b2dc74a4dbea7ae99f5303b1fa8c562fb7533053d65fb"} Nov 03 18:37:16.300301 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:16.300271 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nl6fg" event={"ID":"d0ee858f-ae52-4de9-b576-ed63754072f9","Type":"ContainerStarted","Data":"c2c7bc19b15e2908a69f8c51a9cfbd6ed0be0f3c1f4b65ad81a84f051de60157"} Nov 03 18:37:16.309779 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:16.309749 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"8a249de0ac9079d1551a521e9f3cd5d3","Type":"ContainerStarted","Data":"c6a981367ab924b0f7c87973ad06271afd99971d1f5d20ddbcd28010d4043b2b"} Nov 03 18:37:16.535548 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:16.535487 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hmqxx" podStartSLOduration=1.691117175 podStartE2EDuration="7.535468407s" totalImagesPullingTime="5.844351232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:16.415172267 +0000 UTC m=+8.670160818" watchObservedRunningTime="2026-11-03 18:37:16.535468407 +0000 UTC m=+8.790456959" Nov 03 18:37:16.651919 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:16.651862 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" podStartSLOduration=7.651843166 podStartE2EDuration="7.651843166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:16.534903823 +0000 UTC m=+8.789892378" watchObservedRunningTime="2026-11-03 18:37:16.651843166 +0000 UTC m=+8.906831718" Nov 03 18:37:17.094115 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:17.094033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:17.094273 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:17.094124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:17.094273 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.094251 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:17.094387 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.094309 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:25.094291573 +0000 UTC m=+17.349280116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:17.094387 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.094371 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:17.094500 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.094402 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret podName:8da56f0f-6ab9-4eb3-ad1d-35ce98845ada nodeName:}" failed. No retries permitted until 2026-11-03 18:37:25.094390636 +0000 UTC m=+17.349379181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret") pod "global-pull-secret-syncer-t4dcp" (UID: "8da56f0f-6ab9-4eb3-ad1d-35ce98845ada") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:17.184599 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:17.184565 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:17.184792 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.184715 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:17.184860 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:17.184791 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:17.184904 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.184881 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:17.184939 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:17.184926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:17.185011 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.184992 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:17.194853 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:17.194821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:17.195024 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.195005 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:17.195107 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.195036 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:17.195107 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.195048 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m5fj8 for pod openshift-network-diagnostics/network-check-target-6bn8h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:17.195107 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:17.195103 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8 podName:6b7d3c74-e08e-44b1-ab35-ced1363278f0 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:25.195086934 +0000 UTC m=+17.450075484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5fj8" (UniqueName: "kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8") pod "network-check-target-6bn8h" (UID: "6b7d3c74-e08e-44b1-ab35-ced1363278f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:17.314833 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:17.314800 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" event={"ID":"af92784a-e931-457d-a327-5fbba2f37af5","Type":"ContainerStarted","Data":"044818d7d8962818964ec6bba3ff216c4b7dc3dff4d37faa7b25d986f3f649f5"} Nov 03 18:37:18.837340 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:18.837281 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:37:19.181520 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.180884 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:37:18.837302337Z","UUID":"1bf2045a-f1db-41a3-80a9-c04612209175","Handler":null,"Name":"","Endpoint":""} Nov 03 18:37:19.182436 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.182407 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:37:19.182436 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.182434 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:37:19.183381 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.183358 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:19.183542 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.183507 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:19.183542 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:19.183513 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:19.183542 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.183539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:19.183736 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:19.183638 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:19.183799 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:19.183736 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:19.322107 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.322037 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-64c5w" event={"ID":"1d027ae4-4c9c-4beb-8134-3995b81d32c1","Type":"ContainerStarted","Data":"e93b5125a00e49d3cb28ca979accb2ccafb21451ec4a15138c38b800469ee156"} Nov 03 18:37:19.324453 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.324426 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlhcd" event={"ID":"6319a2b4-76e5-42dc-8097-a52570ed7f2c","Type":"ContainerStarted","Data":"4cd3d473e65adf3448940107be55cd06f70b6a45e5d27133b69ee9f3d4485144"} Nov 03 18:37:19.326835 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.326812 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" event={"ID":"af92784a-e931-457d-a327-5fbba2f37af5","Type":"ContainerStarted","Data":"b624164988e39db19f021818236924cc422353f0c3c2d3ca511a491cb20fa843"} Nov 03 18:37:19.442534 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.442468 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-64c5w" podStartSLOduration=3.436522724 podStartE2EDuration="11.442452242s" totalImagesPullingTime="8.005929518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:19.442095744 +0000 UTC m=+11.697084295" watchObservedRunningTime="2026-11-03 18:37:19.442452242 +0000 UTC m=+11.697440794" Nov 03 18:37:19.443272 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.443235 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nl6fg" podStartSLOduration=6.844602046 podStartE2EDuration="11.443221642s" totalImagesPullingTime="4.598619596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:16.652952028 +0000 UTC m=+8.907940581" watchObservedRunningTime="2026-11-03 18:37:19.443221642 +0000 UTC m=+11.698210194" Nov 03 18:37:19.561125 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.561063 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tlhcd" podStartSLOduration=3.277989399 podStartE2EDuration="11.561042301s" totalImagesPullingTime="8.283052902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:19.560410466 +0000 UTC m=+11.815399018" watchObservedRunningTime="2026-11-03 18:37:19.561042301 +0000 UTC m=+11.816030854" Nov 03 18:37:19.992840 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.992803 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:19.993415 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:19.993386 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:20.331884 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:20.331855 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qdczr" event={"ID":"a59c6f41-01de-40f1-9725-1ed9bd6784d6","Type":"ContainerStarted","Data":"4df9783fcc54dfb0534c6b9ce64bcb690dc2d498d3147510fa403490326d99ca"} Nov 03 18:37:20.333935 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:20.333906 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="0e587888ed4eb7668678eaf773cb284c0f496691dff7977abf868c03f9e695e4" exitCode=0 Nov 03 18:37:20.334825 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:20.334802 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerDied","Data":"0e587888ed4eb7668678eaf773cb284c0f496691dff7977abf868c03f9e695e4"} Nov 03 18:37:20.335811 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:20.335726 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:20.336139 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:20.336117 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:20.448653 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:20.448598 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qdczr" podStartSLOduration=3.156017693 podStartE2EDuration="11.448581462s" totalImagesPullingTime="8.292563769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:20.447829632 +0000 UTC m=+12.702818197" watchObservedRunningTime="2026-11-03 18:37:20.448581462 +0000 UTC m=+12.703570016" Nov 03 18:37:21.183822 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:21.183600 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:21.184252 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:21.183608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:21.184252 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:21.183931 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:21.184252 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:21.183668 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:21.184252 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:21.184068 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:21.184252 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:21.184164 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:21.339480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:21.339046 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" event={"ID":"af92784a-e931-457d-a327-5fbba2f37af5","Type":"ContainerStarted","Data":"96d0d3166dfa34b4adc6189a69f1bd2e973720c1e1a9c95e3cca636f31d4ba57"} Nov 03 18:37:21.340883 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:21.340853 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clxql" event={"ID":"175d5971-984f-4e61-bdaa-f1b34609f0e4","Type":"ContainerStarted","Data":"317e75988d02b6861d2d957ba84e66d0167864a6b47ae209211f4b907ddd7c20"} Nov 03 18:37:21.577461 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:21.577378 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" podStartSLOduration=2.4398708940000002 podStartE2EDuration="12.577363372s" totalImagesPullingTime="10.137492478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:21.458230039 +0000 UTC m=+13.713218590" watchObservedRunningTime="2026-11-03 18:37:21.577363372 +0000 UTC m=+13.832351930" Nov 03 18:37:21.577593 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:21.577471 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-clxql" podStartSLOduration=3.121163476 podStartE2EDuration="13.577467728s" totalImagesPullingTime="10.456304252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:21.577000646 +0000 UTC m=+13.831989198" watchObservedRunningTime="2026-11-03 18:37:21.577467728 +0000 UTC m=+13.832456279" Nov 03 18:37:22.345248 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:22.345154 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="45b0f9931a0d84621b69f10b2e14455b73908241e7ac67707ae94ed7feaf578a" exitCode=0 Nov 03 18:37:22.345893 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:22.345272 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerDied","Data":"45b0f9931a0d84621b69f10b2e14455b73908241e7ac67707ae94ed7feaf578a"} Nov 03 18:37:23.184257 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:23.184223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:23.184257 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:23.184240 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:23.184469 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:23.184391 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:23.184469 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:23.184444 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:23.184579 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:23.184526 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:23.184644 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:23.184628 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:24.351930 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:24.351893 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="253498e11436a9a3941dff8315793a62f4a9bb5c212924efa2d7de5676a32bf8" exitCode=0 Nov 03 18:37:24.352483 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:24.351949 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerDied","Data":"253498e11436a9a3941dff8315793a62f4a9bb5c212924efa2d7de5676a32bf8"} Nov 03 18:37:25.162942 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:25.162903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:25.163127 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:25.162958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:25.163127 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.163086 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:25.163127 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.163105 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:25.163261 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.163158 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:41.163135578 +0000 UTC m=+33.418124119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:25.163261 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.163174 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret podName:8da56f0f-6ab9-4eb3-ad1d-35ce98845ada nodeName:}" failed. No retries permitted until 2026-11-03 18:37:41.163167682 +0000 UTC m=+33.418156211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret") pod "global-pull-secret-syncer-t4dcp" (UID: "8da56f0f-6ab9-4eb3-ad1d-35ce98845ada") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:25.183842 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:25.183820 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:25.183934 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.183900 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:25.183934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:25.183932 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:25.184031 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.183983 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:25.184031 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:25.184014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:25.184096 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.184062 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:25.263626 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:25.263592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:25.263791 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.263773 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:25.263868 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.263799 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:25.263868 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.263815 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m5fj8 for pod openshift-network-diagnostics/network-check-target-6bn8h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:25.263941 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:25.263875 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8 podName:6b7d3c74-e08e-44b1-ab35-ced1363278f0 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:41.263857617 +0000 UTC m=+33.518846161 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5fj8" (UniqueName: "kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8") pod "network-check-target-6bn8h" (UID: "6b7d3c74-e08e-44b1-ab35-ced1363278f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:26.359684 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.359510 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:37:26.360209 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.360061 2582 generic.go:356] "Generic (PLEG): container finished" podID="ce2ce282-8a19-40f3-a97f-a270daedbe66" containerID="94e7e3976e59ce02aae9240f907db86edc530047d91bbf6ef3159f19e919893d" exitCode=1 Nov 03 18:37:26.360209 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.360118 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"65a88d69c2c6560113b3c4ffb34bddc1210f56a0dc6840cab47cfe775f6c2743"} Nov 03 18:37:26.360209 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.360170 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"d3015eb04d62767166e8cff5785c8aab896d6a6628a3d88cb0cb74e12602705c"} Nov 03 18:37:26.360346 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.360216 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"d8b21259927542a85b6c1ea436f6a3c16b1f8b675f9f1f29e2f374c6999f706c"} Nov 03 18:37:26.360346 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.360248 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"da88da6ce3a277e349f9d12a79c3ac349de842a0c68d2eebf3381c4a25ce0e02"} Nov 03 18:37:26.360346 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.360282 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerDied","Data":"94e7e3976e59ce02aae9240f907db86edc530047d91bbf6ef3159f19e919893d"} Nov 03 18:37:26.360346 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:26.360309 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"9bd78461723c9b61793995dd76c740e4370b43f0b4ddeae237df0098e54e1aa1"} Nov 03 18:37:27.183671 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:27.183629 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:27.183671 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:27.183655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:27.183886 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:27.183759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:27.183886 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:27.183763 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:27.183886 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:27.183836 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:27.183966 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:27.183910 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:27.364897 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:27.364867 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="94050adafc46af6a6f7a2e54358f7121e9e95c85fab39a582484a7d86a884eb6" exitCode=0 Nov 03 18:37:27.365246 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:27.364923 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerDied","Data":"94050adafc46af6a6f7a2e54358f7121e9e95c85fab39a582484a7d86a884eb6"} Nov 03 18:37:28.369634 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:28.369557 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="c9606ebfe68cc38d35b2f69baab44d6bbd803a6f2491baa4b2f55de824e5d23f" exitCode=0 Nov 03 18:37:28.370093 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:28.369649 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerDied","Data":"c9606ebfe68cc38d35b2f69baab44d6bbd803a6f2491baa4b2f55de824e5d23f"} Nov 03 18:37:28.372627 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:28.372612 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:37:28.373000 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:28.372981 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"4362d63032fa1c785679740ba76d1f38af909013639b80ef49baee4682aae23d"} Nov 03 18:37:29.183685 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:29.183655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:29.183685 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:29.183689 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:29.183862 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:29.183770 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:29.183862 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:29.183828 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:29.183951 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:29.183901 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:29.183951 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:29.183942 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:29.378513 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:29.378476 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lv47h" event={"ID":"d4c6be41-6db5-4be6-8cac-76b00171f6af","Type":"ContainerStarted","Data":"c006984a97293d55f70b3dd47e7b442a31edde673b9ed14185e8f605c5227de0"} Nov 03 18:37:29.501986 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:29.501892 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lv47h" podStartSLOduration=8.071567938 podStartE2EDuration="21.501880181s" totalImagesPullingTime="13.430312243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:29.501619766 +0000 UTC m=+21.756608317" watchObservedRunningTime="2026-11-03 18:37:29.501880181 +0000 UTC m=+21.756868731" Nov 03 18:37:30.383421 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:30.383353 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:37:30.384384 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:30.383728 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"d23dd915dc49c718c29973729d2584244a7a27b306874c82422fa700f5053ed6"} Nov 03 18:37:30.384384 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:30.383947 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:30.384384 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:30.384127 2582 scope.go:122] "RemoveContainer" containerID="94e7e3976e59ce02aae9240f907db86edc530047d91bbf6ef3159f19e919893d" Nov 03 18:37:30.398980 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:30.398959 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:31.183903 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.183864 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:31.184057 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.183905 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:31.184057 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.183862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:31.184057 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:31.184008 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:31.184179 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:31.184083 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:31.184225 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:31.184193 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:31.389531 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.389503 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:37:31.389980 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.389876 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" event={"ID":"ce2ce282-8a19-40f3-a97f-a270daedbe66","Type":"ContainerStarted","Data":"d5fab1337d7d6392ba643e1ed895a288a17e10a04d6e3d9e4ea1ab724c7dfa2d"} Nov 03 18:37:31.389980 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.389945 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:37:31.390208 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.390193 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:31.404507 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.404482 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:31.574907 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:31.574633 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" podStartSLOduration=7.0788008 podStartE2EDuration="22.574613691s" totalImagesPullingTime="15.495812891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:31.572309353 +0000 UTC m=+23.827297905" watchObservedRunningTime="2026-11-03 18:37:31.574613691 +0000 UTC m=+23.829602242" Nov 03 18:37:32.392434 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:32.392408 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:37:32.585592 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:32.585399 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4dcp"] Nov 03 18:37:32.585748 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:32.585683 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:32.585795 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:32.585778 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:32.585969 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:32.585951 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bn8h"] Nov 03 18:37:32.586056 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:32.586045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:32.586132 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:32.586117 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:32.587972 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:32.587954 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l5g7x"] Nov 03 18:37:32.588055 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:32.588045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:32.588124 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:32.588111 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:33.394205 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:33.394179 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:37:34.184378 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:34.184351 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:34.184378 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:34.184372 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:34.184550 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:34.184353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:34.184550 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:34.184451 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:34.184550 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:34.184524 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:34.184651 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:34.184628 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:36.184296 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:36.184262 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:36.184296 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:36.184289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:36.185040 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:36.184374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:36.185040 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:36.184399 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:36.185040 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:36.184474 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:36.185040 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:36.184537 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:37.828229 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:37.828199 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:37.828651 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:37.828418 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:37:37.839840 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:37.839814 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:38.184676 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.184593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:38.184676 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.184637 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:38.184904 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.184725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:38.184904 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.184732 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5g7x" podUID="c3e13ce6-21d8-4d9c-b589-46699c28b856" Nov 03 18:37:38.184904 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.184805 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bn8h" podUID="6b7d3c74-e08e-44b1-ab35-ced1363278f0" Nov 03 18:37:38.184904 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.184848 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4dcp" podUID="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" Nov 03 18:37:38.577431 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.577364 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeReady" Nov 03 18:37:38.577553 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.577485 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 18:37:38.665310 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.665280 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dks9"] Nov 03 18:37:38.666884 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.666869 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dks9" Nov 03 18:37:38.672726 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.672686 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pfqbv"] Nov 03 18:37:38.674263 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.674242 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d68qq"] Nov 03 18:37:38.674421 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.674402 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" Nov 03 18:37:38.675658 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.675643 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-drzcf"] Nov 03 18:37:38.675788 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.675772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d68qq" Nov 03 18:37:38.677348 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.677330 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:38.693366 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.693347 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph"] Nov 03 18:37:38.695109 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.695093 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx"] Nov 03 18:37:38.695326 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.695311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.697227 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.697212 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf"] Nov 03 18:37:38.697326 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.697311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.699201 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.699186 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh"] Nov 03 18:37:38.699326 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.699309 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.705152 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.705133 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66b9b978fb-5lvgf"] Nov 03 18:37:38.705257 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.705242 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.707432 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.707412 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v5gvn"] Nov 03 18:37:38.707546 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.707477 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.709891 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.709872 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74945cfd6f-j4qs2"] Nov 03 18:37:38.710076 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.710015 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.712257 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.712242 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-qqwhz"] Nov 03 18:37:38.712382 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.712365 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.715258 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.715243 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2"] Nov 03 18:37:38.715413 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.715395 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.717824 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.717810 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7577979dbf-qm2k5"] Nov 03 18:37:38.717944 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.717929 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" Nov 03 18:37:38.720249 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.720234 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wt7n9"] Nov 03 18:37:38.720341 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.720328 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.722891 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.722871 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb"] Nov 03 18:37:38.722988 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.722956 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.723219 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.723204 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.723219 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.723204 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.725576 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.725559 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jphgt"] Nov 03 18:37:38.725689 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.725672 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" Nov 03 18:37:38.728434 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.728417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:38.746538 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.746518 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dks9"] Nov 03 18:37:38.748679 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.748659 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 18:37:38.748952 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.748932 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-77qxk\"" Nov 03 18:37:38.749076 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.748973 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 18:37:38.750158 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.750137 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pfqbv"] Nov 03 18:37:38.762732 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.762690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e50d63ba-d363-48f5-a720-880cd7c895d0-serving-cert\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.762813 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.762786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289c8acb-0083-4ec9-ab45-d5e94d47c490-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.762856 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.762832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.762898 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.762860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f023639-2351-4534-8884-8ffdc05a4144-nginx-conf\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:38.762898 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.762888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddmb4\" (UniqueName: \"kubernetes.io/projected/289c8acb-0083-4ec9-ab45-d5e94d47c490-kube-api-access-ddmb4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.763015 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763000 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5nn\" (UniqueName: \"kubernetes.io/projected/d0f084d2-6540-44ee-ac2b-c8e6df92a050-kube-api-access-mb5nn\") pod \"kas-connection-checker-685895c7c9-pfqbv\" (UID: \"d0f084d2-6540-44ee-ac2b-c8e6df92a050\") " pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" Nov 03 18:37:38.763053 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64zv\" (UniqueName: \"kubernetes.io/projected/4a6b250a-51c5-4415-bd6b-4f1275517fdc-kube-api-access-x64zv\") pod \"kas-connection-checker-685895c7c9-d68qq\" (UID: \"4a6b250a-51c5-4415-bd6b-4f1275517fdc\") " pod="kube-system/kas-connection-checker-685895c7c9-d68qq" Nov 03 18:37:38.763053 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763039 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:38.763126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763068 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkn57\" (UniqueName: \"kubernetes.io/projected/e50d63ba-d363-48f5-a720-880cd7c895d0-kube-api-access-kkn57\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.763126 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64s6\" (UniqueName: \"kubernetes.io/projected/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-kube-api-access-c64s6\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.763190 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289c8acb-0083-4ec9-ab45-d5e94d47c490-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.763254 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e50d63ba-d363-48f5-a720-880cd7c895d0-config\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.763316 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.763355 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.763322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76ms\" (UniqueName: \"kubernetes.io/projected/245116c0-8b38-4fd5-868e-fe39362a2113-kube-api-access-m76ms\") pod \"kas-connection-checker-685895c7c9-7dks9\" (UID: \"245116c0-8b38-4fd5-868e-fe39362a2113\") " pod="kube-system/kas-connection-checker-685895c7c9-7dks9" Nov 03 18:37:38.765335 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.765315 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d68qq"] Nov 03 18:37:38.765939 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.765923 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-drzcf"] Nov 03 18:37:38.793873 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.793751 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph"] Nov 03 18:37:38.800813 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.800789 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx"] Nov 03 18:37:38.801842 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.801826 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v5gvn"] Nov 03 18:37:38.802953 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.802932 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66b9b978fb-5lvgf"] Nov 03 18:37:38.810630 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.810609 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2"] Nov 03 18:37:38.811192 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.811175 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb"] Nov 03 18:37:38.821383 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821363 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.821498 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821383 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.821498 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821391 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 18:37:38.821498 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821473 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.821498 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821479 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 18:37:38.821674 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821507 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.821674 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821518 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 18:37:38.821674 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821616 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.821869 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821742 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.821869 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821784 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.821869 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821823 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 18:37:38.821869 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821827 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.821869 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.821839 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.822120 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822107 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.822181 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822137 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.822181 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822147 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 18:37:38.822181 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822167 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 18:37:38.822181 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822172 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822188 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822205 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822218 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822225 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822236 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822242 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822331 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.822371 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822355 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.822640 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822408 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.822640 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.822456 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:37:38.864517 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.864517 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8cj\" (UniqueName: \"kubernetes.io/projected/edef417a-da49-4ca8-a744-0a3c7569546f-kube-api-access-7b8cj\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f023639-2351-4534-8884-8ffdc05a4144-nginx-conf\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmb4\" (UniqueName: \"kubernetes.io/projected/289c8acb-0083-4ec9-ab45-d5e94d47c490-kube-api-access-ddmb4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864618 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edef417a-da49-4ca8-a744-0a3c7569546f-config-volume\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64s6\" (UniqueName: \"kubernetes.io/projected/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-kube-api-access-c64s6\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqbv\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.864968 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.864954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-tmp\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865039 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfj5x\" (UniqueName: \"kubernetes.io/projected/30383da3-bb10-420f-8f69-587bb4fdacd5-kube-api-access-wfj5x\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5583debd-4dae-4fc4-8506-3a452b179306-cert\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e50d63ba-d363-48f5-a720-880cd7c895d0-config\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.865195 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.865288 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-stats-auth\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.865309 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.365290971 +0000 UTC m=+31.620279515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865344 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f023639-2351-4534-8884-8ffdc05a4144-nginx-conf\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76ms\" (UniqueName: \"kubernetes.io/projected/245116c0-8b38-4fd5-868e-fe39362a2113-kube-api-access-m76ms\") pod \"kas-connection-checker-685895c7c9-7dks9\" (UID: \"245116c0-8b38-4fd5-868e-fe39362a2113\") " pod="kube-system/kas-connection-checker-685895c7c9-7dks9" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbnd\" (UniqueName: \"kubernetes.io/projected/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-kube-api-access-rxbnd\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbh4\" (UniqueName: \"kubernetes.io/projected/5583debd-4dae-4fc4-8506-3a452b179306-kube-api-access-jdbh4\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-snapshots\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7cq\" (UniqueName: \"kubernetes.io/projected/10497765-1e63-4159-870c-ede32ad1b170-kube-api-access-br7cq\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e50d63ba-d363-48f5-a720-880cd7c895d0-serving-cert\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzzp\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865631 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e50d63ba-d363-48f5-a720-880cd7c895d0-config\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.865656 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865639 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-default-certificate\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.865910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5nn\" (UniqueName: \"kubernetes.io/projected/d0f084d2-6540-44ee-ac2b-c8e6df92a050-kube-api-access-mb5nn\") pod \"kas-connection-checker-685895c7c9-pfqbv\" (UID: \"d0f084d2-6540-44ee-ac2b-c8e6df92a050\") " pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64zv\" (UniqueName: \"kubernetes.io/projected/4a6b250a-51c5-4415-bd6b-4f1275517fdc-kube-api-access-x64zv\") pod \"kas-connection-checker-685895c7c9-d68qq\" (UID: \"4a6b250a-51c5-4415-bd6b-4f1275517fdc\") " pod="kube-system/kas-connection-checker-685895c7c9-d68qq" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75b52\" (UniqueName: \"kubernetes.io/projected/07926d6b-bf32-412b-89e5-26c0778a793f-kube-api-access-75b52\") pod \"network-check-source-6f78d59f99-tqhrb\" (UID: \"07926d6b-bf32-412b-89e5-26c0778a793f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f64f0eb-fed7-496b-92a6-84601b0bb544-serving-cert\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289c8acb-0083-4ec9-ab45-d5e94d47c490-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10497765-1e63-4159-870c-ede32ad1b170-serving-cert\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqqt\" (UniqueName: \"kubernetes.io/projected/fe291b57-902f-430e-91ed-d373c931c885-kube-api-access-2jqqt\") pod \"volume-data-source-validator-58c9f5f46d-q8cd2\" (UID: \"fe291b57-902f-430e-91ed-d373c931c885\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" Nov 03 18:37:38.866480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkn57\" (UniqueName: \"kubernetes.io/projected/e50d63ba-d363-48f5-a720-880cd7c895d0-kube-api-access-kkn57\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866493 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edef417a-da49-4ca8-a744-0a3c7569546f-tmp-dir\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866709 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866734 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-config\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866813 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289c8acb-0083-4ec9-ab45-d5e94d47c490-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.866997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289c8acb-0083-4ec9-ab45-d5e94d47c490-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.867029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.867077 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.867154 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.867101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xg7h\" (UniqueName: \"kubernetes.io/projected/8f64f0eb-fed7-496b-92a6-84601b0bb544-kube-api-access-8xg7h\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.869600 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.869567 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e50d63ba-d363-48f5-a720-880cd7c895d0-serving-cert\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.873422 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.873403 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 18:37:38.876670 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.876637 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.902089 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.902068 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 18:37:38.902584 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.902565 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 18:37:38.902643 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.902592 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 18:37:38.903081 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.903067 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wt7n9"] Nov 03 18:37:38.903653 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.903636 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf"] Nov 03 18:37:38.908887 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.908872 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:37:38.908969 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.908951 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 18:37:38.909017 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.908976 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 18:37:38.909209 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909152 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 18:37:38.909209 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909176 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 18:37:38.909209 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909193 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:37:38.909326 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909198 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zjfk4\"" Nov 03 18:37:38.909366 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909323 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 18:37:38.909419 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909378 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j86t5\"" Nov 03 18:37:38.909419 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909380 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p9fdh\"" Nov 03 18:37:38.909419 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909405 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n2c2b\"" Nov 03 18:37:38.909534 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909467 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 18:37:38.909936 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.909921 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74945cfd6f-j4qs2"] Nov 03 18:37:38.911886 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.911855 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc7sj\"" Nov 03 18:37:38.912127 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.912101 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 18:37:38.912127 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.912104 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 18:37:38.912127 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.912123 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 18:37:38.912290 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.912135 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 18:37:38.912290 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.912156 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z9j9b\"" Nov 03 18:37:38.912388 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.912377 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 18:37:38.912440 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.912428 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-78v26\"" Nov 03 18:37:38.913290 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.913276 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh"] Nov 03 18:37:38.913890 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.913878 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jphgt"] Nov 03 18:37:38.914473 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.914459 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-qqwhz"] Nov 03 18:37:38.917497 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.917484 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:38.917551 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.917532 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.417518362 +0000 UTC m=+31.672506896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:38.919490 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.919468 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289c8acb-0083-4ec9-ab45-d5e94d47c490-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.923995 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.923977 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7577979dbf-qm2k5"] Nov 03 18:37:38.931918 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.931887 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkn57\" (UniqueName: \"kubernetes.io/projected/e50d63ba-d363-48f5-a720-880cd7c895d0-kube-api-access-kkn57\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:38.931918 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.931902 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmb4\" (UniqueName: \"kubernetes.io/projected/289c8acb-0083-4ec9-ab45-d5e94d47c490-kube-api-access-ddmb4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:38.932066 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.931898 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64zv\" (UniqueName: \"kubernetes.io/projected/4a6b250a-51c5-4415-bd6b-4f1275517fdc-kube-api-access-x64zv\") pod \"kas-connection-checker-685895c7c9-d68qq\" (UID: \"4a6b250a-51c5-4415-bd6b-4f1275517fdc\") " pod="kube-system/kas-connection-checker-685895c7c9-d68qq" Nov 03 18:37:38.932121 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.932072 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5nn\" (UniqueName: \"kubernetes.io/projected/d0f084d2-6540-44ee-ac2b-c8e6df92a050-kube-api-access-mb5nn\") pod \"kas-connection-checker-685895c7c9-pfqbv\" (UID: \"d0f084d2-6540-44ee-ac2b-c8e6df92a050\") " pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" Nov 03 18:37:38.932337 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.932322 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64s6\" (UniqueName: \"kubernetes.io/projected/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-kube-api-access-c64s6\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:38.933599 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.933584 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76ms\" (UniqueName: \"kubernetes.io/projected/245116c0-8b38-4fd5-868e-fe39362a2113-kube-api-access-m76ms\") pod \"kas-connection-checker-685895c7c9-7dks9\" (UID: \"245116c0-8b38-4fd5-868e-fe39362a2113\") " pod="kube-system/kas-connection-checker-685895c7c9-7dks9" Nov 03 18:37:38.967685 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.967790 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.967790 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg7h\" (UniqueName: \"kubernetes.io/projected/8f64f0eb-fed7-496b-92a6-84601b0bb544-kube-api-access-8xg7h\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.967790 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.967790 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967782 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8cj\" (UniqueName: \"kubernetes.io/projected/edef417a-da49-4ca8-a744-0a3c7569546f-kube-api-access-7b8cj\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.967792 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.967805 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edef417a-da49-4ca8-a744-0a3c7569546f-config-volume\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.967855 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.467839598 +0000 UTC m=+31.722828143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqbv\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.967959 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-tmp\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.967972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfj5x\" (UniqueName: \"kubernetes.io/projected/30383da3-bb10-420f-8f69-587bb4fdacd5-kube-api-access-wfj5x\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5583debd-4dae-4fc4-8506-3a452b179306-cert\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968071 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-stats-auth\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbnd\" (UniqueName: \"kubernetes.io/projected/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-kube-api-access-rxbnd\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbh4\" (UniqueName: \"kubernetes.io/projected/5583debd-4dae-4fc4-8506-3a452b179306-kube-api-access-jdbh4\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968202 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-snapshots\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7cq\" (UniqueName: \"kubernetes.io/projected/10497765-1e63-4159-870c-ede32ad1b170-kube-api-access-br7cq\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzzp\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.968315 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-default-certificate\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968421 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-tmp\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968465 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edef417a-da49-4ca8-a744-0a3c7569546f-config-volume\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.968546 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.968605 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.46859036 +0000 UTC m=+31.723578890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b52\" (UniqueName: \"kubernetes.io/projected/07926d6b-bf32-412b-89e5-26c0778a793f-kube-api-access-75b52\") pod \"network-check-source-6f78d59f99-tqhrb\" (UID: \"07926d6b-bf32-412b-89e5-26c0778a793f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968737 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f64f0eb-fed7-496b-92a6-84601b0bb544-serving-cert\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10497765-1e63-4159-870c-ede32ad1b170-serving-cert\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968798 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968804 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqqt\" (UniqueName: \"kubernetes.io/projected/fe291b57-902f-430e-91ed-d373c931c885-kube-api-access-2jqqt\") pod \"volume-data-source-validator-58c9f5f46d-q8cd2\" (UID: \"fe291b57-902f-430e-91ed-d373c931c885\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" Nov 03 18:37:38.968934 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968917 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968939 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.968948 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edef417a-da49-4ca8-a744-0a3c7569546f-tmp-dir\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969001 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969203 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.969642 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969301 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.970055 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.970055 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.970055 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:38.970055 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-config\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.970055 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.969916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.970290 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.970130 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.970290 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.970233 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.470217688 +0000 UTC m=+31.725206246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:38.970491 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.970475 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:38.970597 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.970585 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:38.970756 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.970743 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.470725942 +0000 UTC m=+31.725714482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:38.970921 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.970896 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5583debd-4dae-4fc4-8506-3a452b179306-cert\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:38.971374 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.971354 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edef417a-da49-4ca8-a744-0a3c7569546f-tmp-dir\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:38.971516 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.971405 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10497765-1e63-4159-870c-ede32ad1b170-serving-cert\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.972083 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.971847 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f64f0eb-fed7-496b-92a6-84601b0bb544-serving-cert\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.972083 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.971993 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-stats-auth\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.972083 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.970529 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.972083 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.972040 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-snapshots\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:38.972307 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.972287 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.972385 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.972368 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:38.972444 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.972418 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.972444 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.972430 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.472414881 +0000 UTC m=+31.727403427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:38.972551 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.972493 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:38.972637 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:38.972624 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:39.472611991 +0000 UTC m=+31.727600533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:38.973037 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.973011 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.973148 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.973130 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-config\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:38.973201 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.973164 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.973811 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.973791 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:38.974214 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.974197 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:38.974692 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.974674 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-default-certificate\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:38.980586 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.979864 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dks9" Nov 03 18:37:38.985419 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.985404 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" Nov 03 18:37:38.991127 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:38.991109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d68qq" Nov 03 18:37:39.009002 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.008967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:39.016883 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.015393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:39.037825 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.036093 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:39.040500 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.040341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8cj\" (UniqueName: \"kubernetes.io/projected/edef417a-da49-4ca8-a744-0a3c7569546f-kube-api-access-7b8cj\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:39.045376 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.045306 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfj5x\" (UniqueName: \"kubernetes.io/projected/30383da3-bb10-420f-8f69-587bb4fdacd5-kube-api-access-wfj5x\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:39.046174 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.046152 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7cq\" (UniqueName: \"kubernetes.io/projected/10497765-1e63-4159-870c-ede32ad1b170-kube-api-access-br7cq\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:39.047655 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.046934 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbh4\" (UniqueName: \"kubernetes.io/projected/5583debd-4dae-4fc4-8506-3a452b179306-kube-api-access-jdbh4\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:39.047655 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.046951 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqqt\" (UniqueName: \"kubernetes.io/projected/fe291b57-902f-430e-91ed-d373c931c885-kube-api-access-2jqqt\") pod \"volume-data-source-validator-58c9f5f46d-q8cd2\" (UID: \"fe291b57-902f-430e-91ed-d373c931c885\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" Nov 03 18:37:39.047655 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.047404 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:39.047655 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.047561 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzzp\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:39.048059 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.047968 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b52\" (UniqueName: \"kubernetes.io/projected/07926d6b-bf32-412b-89e5-26c0778a793f-kube-api-access-75b52\") pod \"network-check-source-6f78d59f99-tqhrb\" (UID: \"07926d6b-bf32-412b-89e5-26c0778a793f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" Nov 03 18:37:39.048758 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.048689 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqbv\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:39.048851 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.048767 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg7h\" (UniqueName: \"kubernetes.io/projected/8f64f0eb-fed7-496b-92a6-84601b0bb544-kube-api-access-8xg7h\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:39.050987 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.050924 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbnd\" (UniqueName: \"kubernetes.io/projected/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-kube-api-access-rxbnd\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:39.053495 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.053317 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:39.061756 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.061576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" Nov 03 18:37:39.077631 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.077416 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" Nov 03 18:37:39.082727 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.082516 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:39.341482 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.341456 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:39.376899 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.376765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:39.377037 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.377000 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:39.377098 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.377081 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.377060161 +0000 UTC m=+32.632048716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:39.381906 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.381885 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dks9"] Nov 03 18:37:39.385013 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.384981 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pfqbv"] Nov 03 18:37:39.387647 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:39.387606 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245116c0_8b38_4fd5_868e_fe39362a2113.slice/crio-19921ad23950c57880563672898b2bda149c771febc02109a741231fae0ba973 WatchSource:0}: Error finding container 19921ad23950c57880563672898b2bda149c771febc02109a741231fae0ba973: Status 404 returned error can't find the container with id 19921ad23950c57880563672898b2bda149c771febc02109a741231fae0ba973 Nov 03 18:37:39.389503 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.388790 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d68qq"] Nov 03 18:37:39.390241 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:39.389773 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f084d2_6540_44ee_ac2b_c8e6df92a050.slice/crio-c3c460fd9e24abe8d65a62a828d63c0598577e9f7837b89e2e05416270602027 WatchSource:0}: Error finding container c3c460fd9e24abe8d65a62a828d63c0598577e9f7837b89e2e05416270602027: Status 404 returned error can't find the container with id c3c460fd9e24abe8d65a62a828d63c0598577e9f7837b89e2e05416270602027 Nov 03 18:37:39.415042 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.414976 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d68qq" event={"ID":"4a6b250a-51c5-4415-bd6b-4f1275517fdc","Type":"ContainerStarted","Data":"3b6d1b16d41588f5ed954db7693a192a86884bca7821cda6b90d55486e9a915a"} Nov 03 18:37:39.419226 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.419196 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" event={"ID":"d0f084d2-6540-44ee-ac2b-c8e6df92a050","Type":"ContainerStarted","Data":"c3c460fd9e24abe8d65a62a828d63c0598577e9f7837b89e2e05416270602027"} Nov 03 18:37:39.420570 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.420542 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dks9" event={"ID":"245116c0-8b38-4fd5-868e-fe39362a2113","Type":"ContainerStarted","Data":"19921ad23950c57880563672898b2bda149c771febc02109a741231fae0ba973"} Nov 03 18:37:39.431523 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.431499 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph"] Nov 03 18:37:39.435292 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.435225 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx"] Nov 03 18:37:39.448254 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:39.448221 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289c8acb_0083_4ec9_ab45_d5e94d47c490.slice/crio-08226d1e6a9a46d050f1d4ba28ad63976d864f51e1fd5094459bbeca87410150 WatchSource:0}: Error finding container 08226d1e6a9a46d050f1d4ba28ad63976d864f51e1fd5094459bbeca87410150: Status 404 returned error can't find the container with id 08226d1e6a9a46d050f1d4ba28ad63976d864f51e1fd5094459bbeca87410150 Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.477540 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.477587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.477618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.477713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.477747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.477790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.477874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478011 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478024 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478073 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.478055713 +0000 UTC m=+32.733044259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478140 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478174 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.478163745 +0000 UTC m=+32.733152288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478260 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.478250269 +0000 UTC m=+32.733238811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478320 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478348 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.478339231 +0000 UTC m=+32.733327775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:39.478654 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478406 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:39.479458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478436 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.478426834 +0000 UTC m=+32.733415380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:39.479458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478487 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:39.479458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478526 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.478516444 +0000 UTC m=+32.733504987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:39.479458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478585 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:39.479458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478595 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:39.479458 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:39.478623 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.47861415 +0000 UTC m=+32.733602692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:39.505895 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.504728 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-qqwhz"] Nov 03 18:37:39.510999 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.510977 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb"] Nov 03 18:37:39.511325 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:39.511301 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10497765_1e63_4159_870c_ede32ad1b170.slice/crio-229e4f2c779762cb21e78cb31a9054c48f1d1bc11d4f4fe93c1f7826e9b3653f WatchSource:0}: Error finding container 229e4f2c779762cb21e78cb31a9054c48f1d1bc11d4f4fe93c1f7826e9b3653f: Status 404 returned error can't find the container with id 229e4f2c779762cb21e78cb31a9054c48f1d1bc11d4f4fe93c1f7826e9b3653f Nov 03 18:37:39.516766 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.516402 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2"] Nov 03 18:37:39.517757 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.517550 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jphgt"] Nov 03 18:37:39.524528 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:39.523804 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe291b57_902f_430e_91ed_d373c931c885.slice/crio-9dc945b69a3be9076725fcd61450e852f1ab634ac66ed32f5d17d6a8ebbdc9ef WatchSource:0}: Error finding container 9dc945b69a3be9076725fcd61450e852f1ab634ac66ed32f5d17d6a8ebbdc9ef: Status 404 returned error can't find the container with id 9dc945b69a3be9076725fcd61450e852f1ab634ac66ed32f5d17d6a8ebbdc9ef Nov 03 18:37:39.712532 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:39.712460 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v5gvn"] Nov 03 18:37:39.715105 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:39.715074 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f64f0eb_fed7_496b_92a6_84601b0bb544.slice/crio-069dc0958b2c876b35b84e5ab0dc7d41c3d0e4c6dc4fc6000869ae7062d4af6b WatchSource:0}: Error finding container 069dc0958b2c876b35b84e5ab0dc7d41c3d0e4c6dc4fc6000869ae7062d4af6b: Status 404 returned error can't find the container with id 069dc0958b2c876b35b84e5ab0dc7d41c3d0e4c6dc4fc6000869ae7062d4af6b Nov 03 18:37:40.183503 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.183423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:40.183503 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.183440 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:40.183990 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.183423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:40.238749 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.238720 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:37:40.238889 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.238826 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkfh8\"" Nov 03 18:37:40.238889 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.238867 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:37:40.238975 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.238897 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dvl9\"" Nov 03 18:37:40.238975 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.238901 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4grw\"" Nov 03 18:37:40.385372 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.385338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:40.385640 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.385496 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:40.385770 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.385716 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.385680425 +0000 UTC m=+34.640668957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:40.424767 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.424734 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dks9" event={"ID":"245116c0-8b38-4fd5-868e-fe39362a2113","Type":"ContainerStarted","Data":"5f10ee396c98ade79c78c5c32306ada56c5846b5174544c1b8c952cf3f6be02f"} Nov 03 18:37:40.425954 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.425928 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" event={"ID":"8f64f0eb-fed7-496b-92a6-84601b0bb544","Type":"ContainerStarted","Data":"069dc0958b2c876b35b84e5ab0dc7d41c3d0e4c6dc4fc6000869ae7062d4af6b"} Nov 03 18:37:40.427480 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.427457 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" event={"ID":"fe291b57-902f-430e-91ed-d373c931c885","Type":"ContainerStarted","Data":"9dc945b69a3be9076725fcd61450e852f1ab634ac66ed32f5d17d6a8ebbdc9ef"} Nov 03 18:37:40.428833 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.428807 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqwhz" event={"ID":"10497765-1e63-4159-870c-ede32ad1b170","Type":"ContainerStarted","Data":"229e4f2c779762cb21e78cb31a9054c48f1d1bc11d4f4fe93c1f7826e9b3653f"} Nov 03 18:37:40.430283 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.430260 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" event={"ID":"289c8acb-0083-4ec9-ab45-d5e94d47c490","Type":"ContainerStarted","Data":"08226d1e6a9a46d050f1d4ba28ad63976d864f51e1fd5094459bbeca87410150"} Nov 03 18:37:40.431585 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.431558 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" event={"ID":"e50d63ba-d363-48f5-a720-880cd7c895d0","Type":"ContainerStarted","Data":"0387d9bbc097c437da6d05f4b2d13fe60e5df5490bc0d80f014cee1aaec41db3"} Nov 03 18:37:40.433023 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.432998 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jphgt" event={"ID":"5583debd-4dae-4fc4-8506-3a452b179306","Type":"ContainerStarted","Data":"77f30296d3aaba6d9b86c4b006ba68536728dd86919b0df0f40d1e4f283392d8"} Nov 03 18:37:40.435902 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.435370 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d68qq" event={"ID":"4a6b250a-51c5-4415-bd6b-4f1275517fdc","Type":"ContainerStarted","Data":"e568a271e83dc8cae60dea5848b4b28cfe91f74bb145f5d286d23fa67027c7e8"} Nov 03 18:37:40.438626 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.438168 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" event={"ID":"d0f084d2-6540-44ee-ac2b-c8e6df92a050","Type":"ContainerStarted","Data":"a56a4b15688258c51ab6c57a7e8a2b0cf7718efdd2745ec70b78cfba4369f50e"} Nov 03 18:37:40.441169 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.441146 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" event={"ID":"07926d6b-bf32-412b-89e5-26c0778a793f","Type":"ContainerStarted","Data":"4efec120778c48bafdef06b5ffd62e8b2f7ab4b39e4e85630938773f420a7d56"} Nov 03 18:37:40.487068 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.487036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:40.487222 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.487097 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:40.487222 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.487124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:40.487222 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.487144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:40.487222 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.487208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:40.487433 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.487230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:40.487433 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:40.487263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:40.487433 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487363 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:40.487433 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487377 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:40.487433 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487419 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.487406701 +0000 UTC m=+34.742395229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:40.487681 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487470 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:40.487681 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487506 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:40.487681 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487530 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.487522823 +0000 UTC m=+34.742511365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:40.487681 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487580 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:40.487681 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487603 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.487593953 +0000 UTC m=+34.742582496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:40.487937 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487689 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.487682957 +0000 UTC m=+34.742671499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:40.487937 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487758 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:40.487937 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487785 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.487778908 +0000 UTC m=+34.742767449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:40.487937 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487825 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:40.487937 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487844 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.487837753 +0000 UTC m=+34.742826295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:40.487937 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487883 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:40.487937 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:40.487908 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:42.487901816 +0000 UTC m=+34.742890346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:41.049534 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.047981 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-d68qq" podStartSLOduration=391.047963139 podStartE2EDuration="6m31.047963139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:41.047059799 +0000 UTC m=+33.302048349" watchObservedRunningTime="2026-11-03 18:37:41.047963139 +0000 UTC m=+33.302951690" Nov 03 18:37:41.049534 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.048125 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" podStartSLOduration=391.048119926 podStartE2EDuration="6m31.048119926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:40.930411544 +0000 UTC m=+33.185400095" watchObservedRunningTime="2026-11-03 18:37:41.048119926 +0000 UTC m=+33.303108504" Nov 03 18:37:41.166788 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.165888 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7dks9" podStartSLOduration=391.165869165 podStartE2EDuration="6m31.165869165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:41.164088923 +0000 UTC m=+33.419077476" watchObservedRunningTime="2026-11-03 18:37:41.165869165 +0000 UTC m=+33.420857721" Nov 03 18:37:41.195812 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:41.195047 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 18:37:41.195812 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.195121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:41.195812 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.195206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:41.195812 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:41.195789 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:13.195769936 +0000 UTC m=+65.450758464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : secret "metrics-daemon-secret" not found Nov 03 18:37:41.197912 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.197863 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:41.296663 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.296612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:41.302297 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.302234 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:41.396749 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.396717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:41.401309 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.401278 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:41.817264 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.817034 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bn8h"] Nov 03 18:37:41.822049 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:41.822016 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7d3c74_e08e_44b1_ab35_ced1363278f0.slice/crio-752ae2fa5ac98390145f5e9b63d8c6bc9f01c0ddac76235ca3a29aab204f2ad8 WatchSource:0}: Error finding container 752ae2fa5ac98390145f5e9b63d8c6bc9f01c0ddac76235ca3a29aab204f2ad8: Status 404 returned error can't find the container with id 752ae2fa5ac98390145f5e9b63d8c6bc9f01c0ddac76235ca3a29aab204f2ad8 Nov 03 18:37:41.841334 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:41.840181 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4dcp"] Nov 03 18:37:41.847834 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:41.847806 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da56f0f_6ab9_4eb3_ad1d_35ce98845ada.slice/crio-406ee152fba5738bd51f49f57dc4923d333e334d790aa87bbf211050f82d1af0 WatchSource:0}: Error finding container 406ee152fba5738bd51f49f57dc4923d333e334d790aa87bbf211050f82d1af0: Status 404 returned error can't find the container with id 406ee152fba5738bd51f49f57dc4923d333e334d790aa87bbf211050f82d1af0 Nov 03 18:37:42.408054 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.407171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:42.408054 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.407505 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:42.408054 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.407580 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.407560509 +0000 UTC m=+38.662549053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:42.458764 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.458623 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" event={"ID":"fe291b57-902f-430e-91ed-d373c931c885","Type":"ContainerStarted","Data":"8cf5b65da3df26420d09f0c2f3fa3f8db4b35a1b43bafb5ae927ec6091f5fe38"} Nov 03 18:37:42.460250 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.460208 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bn8h" event={"ID":"6b7d3c74-e08e-44b1-ab35-ced1363278f0","Type":"ContainerStarted","Data":"752ae2fa5ac98390145f5e9b63d8c6bc9f01c0ddac76235ca3a29aab204f2ad8"} Nov 03 18:37:42.461687 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.461638 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t4dcp" event={"ID":"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada","Type":"ContainerStarted","Data":"406ee152fba5738bd51f49f57dc4923d333e334d790aa87bbf211050f82d1af0"} Nov 03 18:37:42.508414 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.508376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:42.508572 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.508437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:42.508572 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.508517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:42.508572 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.508558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:42.508677 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.508586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:42.508677 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.508609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:42.508772 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.508674 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:42.508815 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.508808 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.508863 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.508851045 +0000 UTC m=+38.763839577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.508971 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.508985 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509016 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.508988 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.508970772 +0000 UTC m=+38.763959313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509039 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509050 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509057 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.5090401 +0000 UTC m=+38.764028636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509072 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:42.509081 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509089 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:42.509734 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509076 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.509064983 +0000 UTC m=+38.764053516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:42.509734 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509139 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.50913078 +0000 UTC m=+38.764119312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:42.509734 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509155 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.509145508 +0000 UTC m=+38.764134038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:42.509734 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:42.509171 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:46.50916207 +0000 UTC m=+38.764150600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:42.578185 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:42.577670 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" podStartSLOduration=383.178400685 podStartE2EDuration="6m25.577653793s" totalImagesPullingTime="2.399253108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:42.575953225 +0000 UTC m=+34.830941776" watchObservedRunningTime="2026-11-03 18:37:42.577653793 +0000 UTC m=+34.832642347" Nov 03 18:37:45.483585 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.483562 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/0.log" Nov 03 18:37:45.484006 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.483600 2582 generic.go:356] "Generic (PLEG): container finished" podID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerID="1c273a7116f87ac9af7a0ee74d1a14ed1aef0d41e26d2afb1dbdcb5bce4ec455" exitCode=255 Nov 03 18:37:45.484006 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.483668 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" event={"ID":"8f64f0eb-fed7-496b-92a6-84601b0bb544","Type":"ContainerDied","Data":"1c273a7116f87ac9af7a0ee74d1a14ed1aef0d41e26d2afb1dbdcb5bce4ec455"} Nov 03 18:37:45.484006 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.483926 2582 scope.go:122] "RemoveContainer" containerID="1c273a7116f87ac9af7a0ee74d1a14ed1aef0d41e26d2afb1dbdcb5bce4ec455" Nov 03 18:37:45.497504 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.497477 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqwhz" event={"ID":"10497765-1e63-4159-870c-ede32ad1b170","Type":"ContainerStarted","Data":"70bc35c3478ac05b60e28f836e0097fe7219e0cb0636dce79974f4c7d9b41c82"} Nov 03 18:37:45.500633 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.500607 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" event={"ID":"289c8acb-0083-4ec9-ab45-d5e94d47c490","Type":"ContainerStarted","Data":"cef5d39135df21ca14b30be086896158507c77bbfa4aa7550843b9ad419b9292"} Nov 03 18:37:45.502949 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.502925 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" event={"ID":"e50d63ba-d363-48f5-a720-880cd7c895d0","Type":"ContainerStarted","Data":"1c4c256c55842a6ec7f2b709e70a1a2bac8eb381599f5d17bbdcaf2e1f647630"} Nov 03 18:37:45.730281 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.730208 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" podStartSLOduration=385.070272431 podStartE2EDuration="6m30.730186675s" totalImagesPullingTime="5.659914244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:45.725444797 +0000 UTC m=+37.980433351" watchObservedRunningTime="2026-11-03 18:37:45.730186675 +0000 UTC m=+37.985175227" Nov 03 18:37:45.843616 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.843555 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" podStartSLOduration=385.195715268 podStartE2EDuration="6m30.843536654s" totalImagesPullingTime="5.647821386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:45.842431194 +0000 UTC m=+38.097419756" watchObservedRunningTime="2026-11-03 18:37:45.843536654 +0000 UTC m=+38.098525202" Nov 03 18:37:45.964600 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:45.964542 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-qqwhz" podStartSLOduration=383.331248785 podStartE2EDuration="6m28.96452277s" totalImagesPullingTime="5.633273985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:45.962805174 +0000 UTC m=+38.217793726" watchObservedRunningTime="2026-11-03 18:37:45.96452277 +0000 UTC m=+38.219511322" Nov 03 18:37:46.450088 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.450051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:46.450271 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.450243 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:46.450336 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.450325 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.450302865 +0000 UTC m=+46.705291440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:46.509149 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.509120 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/1.log" Nov 03 18:37:46.509608 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.509561 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/0.log" Nov 03 18:37:46.509608 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.509588 2582 generic.go:356] "Generic (PLEG): container finished" podID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerID="9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb" exitCode=255 Nov 03 18:37:46.509795 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.509770 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" event={"ID":"8f64f0eb-fed7-496b-92a6-84601b0bb544","Type":"ContainerDied","Data":"9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb"} Nov 03 18:37:46.509871 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.509839 2582 scope.go:122] "RemoveContainer" containerID="1c273a7116f87ac9af7a0ee74d1a14ed1aef0d41e26d2afb1dbdcb5bce4ec455" Nov 03 18:37:46.509932 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.509921 2582 scope.go:122] "RemoveContainer" containerID="9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb" Nov 03 18:37:46.510103 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.510083 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:37:46.512545 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.512114 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jphgt" event={"ID":"5583debd-4dae-4fc4-8506-3a452b179306","Type":"ContainerStarted","Data":"9cdfcc18f8ff38ded163fb5a406c0ede623b37e67db37d396ed71c7baafc7898"} Nov 03 18:37:46.514389 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.514341 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" event={"ID":"07926d6b-bf32-412b-89e5-26c0778a793f","Type":"ContainerStarted","Data":"fd2e8992d054fcc7740117db929b2adb06b3f03997d745f397444c20826ddccb"} Nov 03 18:37:46.551253 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.551224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:46.551358 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.551275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:46.551358 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.551297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:46.551358 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.551314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:46.551463 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.551372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:46.551463 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551386 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:46.551463 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.551403 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:46.551463 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.551438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:46.551645 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551407 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:46.551733 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551500 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.55148164 +0000 UTC m=+46.806470196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:46.551733 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551677 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:46.551733 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551709 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.551680795 +0000 UTC m=+46.806669342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:46.551733 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551570 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:46.551733 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551725 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551738 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.551725976 +0000 UTC m=+46.806714536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551755 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.551745394 +0000 UTC m=+46.806733924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551570 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551787 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.551777154 +0000 UTC m=+46.806765688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551625 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551822 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.551812918 +0000 UTC m=+46.806801449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551630 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:46.551948 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.551889 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.551872508 +0000 UTC m=+46.806861044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:46.693762 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.693731 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g96rr"] Nov 03 18:37:46.726650 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.726581 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.749155 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.749104 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" podStartSLOduration=352.406917613 podStartE2EDuration="5m58.74908885s" totalImagesPullingTime="6.342171237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:46.748842661 +0000 UTC m=+39.003831215" watchObservedRunningTime="2026-11-03 18:37:46.74908885 +0000 UTC m=+39.004077403" Nov 03 18:37:46.782945 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.782857 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g96rr"] Nov 03 18:37:46.787606 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.787582 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:37:46.787606 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.787582 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88hbk\"" Nov 03 18:37:46.826461 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.826432 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:37:46.855307 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.855269 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvpg\" (UniqueName: \"kubernetes.io/projected/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-api-access-bcvpg\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.855523 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.855482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e6c6d42-58af-440c-8c07-915523a8a3bb-data-volume\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.855646 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.855580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.855738 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.855665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.855811 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.855737 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c6d42-58af-440c-8c07-915523a8a3bb-crio-socket\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.872784 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.872740 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jphgt" podStartSLOduration=2.930525319 podStartE2EDuration="8.872728664s" totalImagesPullingTime="5.942203345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:46.872164818 +0000 UTC m=+39.127153371" watchObservedRunningTime="2026-11-03 18:37:46.872728664 +0000 UTC m=+39.127717212" Nov 03 18:37:46.956336 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.956295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e6c6d42-58af-440c-8c07-915523a8a3bb-data-volume\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.956472 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.956351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.956472 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.956397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.956472 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.956456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c6d42-58af-440c-8c07-915523a8a3bb-crio-socket\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.956581 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.956484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvpg\" (UniqueName: \"kubernetes.io/projected/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-api-access-bcvpg\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.956581 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.956512 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:37:46.956637 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.956584 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:47.456564394 +0000 UTC m=+39.711552940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : secret "insights-runtime-extractor-tls" not found Nov 03 18:37:46.956682 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.956637 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 18:37:46.956682 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.956658 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e6c6d42-58af-440c-8c07-915523a8a3bb-data-volume\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:46.956682 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:46.956670 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:47.456659792 +0000 UTC m=+39.711648334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : configmap "kube-rbac-proxy" not found Nov 03 18:37:46.956794 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:46.956750 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c6d42-58af-440c-8c07-915523a8a3bb-crio-socket\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:47.029222 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.029157 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvpg\" (UniqueName: \"kubernetes.io/projected/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-api-access-bcvpg\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:47.058993 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.058610 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62"] Nov 03 18:37:47.076815 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.076795 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" Nov 03 18:37:47.117012 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.116984 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62"] Nov 03 18:37:47.130952 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.130923 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bgcqc\"" Nov 03 18:37:47.144227 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.144204 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 18:37:47.144347 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.144248 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:47.259831 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.259799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv95g\" (UniqueName: \"kubernetes.io/projected/066c2936-7cba-4b26-8f35-ba18b055aef2-kube-api-access-jv95g\") pod \"migrator-66df5d496b-ztr62\" (UID: \"066c2936-7cba-4b26-8f35-ba18b055aef2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" Nov 03 18:37:47.361147 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.361076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv95g\" (UniqueName: \"kubernetes.io/projected/066c2936-7cba-4b26-8f35-ba18b055aef2-kube-api-access-jv95g\") pod \"migrator-66df5d496b-ztr62\" (UID: \"066c2936-7cba-4b26-8f35-ba18b055aef2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" Nov 03 18:37:47.424901 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.424876 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv95g\" (UniqueName: \"kubernetes.io/projected/066c2936-7cba-4b26-8f35-ba18b055aef2-kube-api-access-jv95g\") pod \"migrator-66df5d496b-ztr62\" (UID: \"066c2936-7cba-4b26-8f35-ba18b055aef2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" Nov 03 18:37:47.462452 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.462412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:47.462621 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.462472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:47.462621 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:47.462563 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 18:37:47.462621 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:47.462577 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:37:47.462621 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:47.462613 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:48.462601343 +0000 UTC m=+40.717589872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : configmap "kube-rbac-proxy" not found Nov 03 18:37:47.462779 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:47.462627 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:48.462620488 +0000 UTC m=+40.717609016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : secret "insights-runtime-extractor-tls" not found Nov 03 18:37:47.518972 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.518949 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/1.log" Nov 03 18:37:47.519408 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.519369 2582 scope.go:122] "RemoveContainer" containerID="9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb" Nov 03 18:37:47.519594 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:47.519567 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:37:47.524151 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.521550 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bn8h" event={"ID":"6b7d3c74-e08e-44b1-ab35-ced1363278f0","Type":"ContainerStarted","Data":"0d46df21dbeb2c10f632e4d7475e6192c27e261a38a01c61920a510363c1168d"} Nov 03 18:37:47.524151 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.522231 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:47.716050 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.716026 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" Nov 03 18:37:47.755533 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:47.755484 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6bn8h" podStartSLOduration=34.543658933 podStartE2EDuration="39.75546738s" totalImagesPullingTime="5.211808447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:47.75485476 +0000 UTC m=+40.009843310" watchObservedRunningTime="2026-11-03 18:37:47.75546738 +0000 UTC m=+40.010455931" Nov 03 18:37:48.103096 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:48.103063 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62"] Nov 03 18:37:48.104850 ip-10-0-128-95 kubenswrapper[2582]: W1103 18:37:48.104820 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066c2936_7cba_4b26_8f35_ba18b055aef2.slice/crio-6e2dabf0642e933e3d4657c2df5e794f5b91e4d5300346d1fa08c6edf2bbb811 WatchSource:0}: Error finding container 6e2dabf0642e933e3d4657c2df5e794f5b91e4d5300346d1fa08c6edf2bbb811: Status 404 returned error can't find the container with id 6e2dabf0642e933e3d4657c2df5e794f5b91e4d5300346d1fa08c6edf2bbb811 Nov 03 18:37:48.473859 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:48.473827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:48.474034 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:48.473890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:48.474034 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:48.474000 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:37:48.474132 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:48.474067 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:50.474051423 +0000 UTC m=+42.729039953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : secret "insights-runtime-extractor-tls" not found Nov 03 18:37:48.474492 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:48.474458 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:48.526112 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:48.526072 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" event={"ID":"066c2936-7cba-4b26-8f35-ba18b055aef2","Type":"ContainerStarted","Data":"6e2dabf0642e933e3d4657c2df5e794f5b91e4d5300346d1fa08c6edf2bbb811"} Nov 03 18:37:49.341640 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:49.341597 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:49.341815 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:49.341671 2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:49.342153 ip-10-0-128-95 kubenswrapper[2582]: I1103 18:37:49.342136 2582 scope.go:122] "RemoveContainer" containerID="9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb" Nov 03 18:37:49.342402 ip-10-0-128-95 kubenswrapper[2582]: E1103 18:37:49.342377 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:37:50.439880 ip-10-0-128-95 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 18:37:50.454132 ip-10-0-128-95 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 18:37:50.454308 ip-10-0-128-95 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 18:37:50.454358 ip-10-0-128-95 systemd[1]: kubelet.service: Consumed 3.035s CPU time. Nov 03 18:37:50.472402 ip-10-0-128-95 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:37:50.596498 ip-10-0-128-95 kubenswrapper[4948]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:50.596498 ip-10-0-128-95 kubenswrapper[4948]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:37:50.596498 ip-10-0-128-95 kubenswrapper[4948]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:50.596938 ip-10-0-128-95 kubenswrapper[4948]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:50.598661 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598570 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.598661 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598660 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598664 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598668 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598671 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598674 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598677 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598680 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598683 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598686 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598689 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598692 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598694 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598713 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598718 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598723 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598727 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598732 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598736 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598741 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.598745 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598745 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598756 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598759 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598762 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598765 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598767 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598770 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598773 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598775 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598778 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598780 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598783 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598786 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598788 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598791 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598793 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598796 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598799 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598801 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.599289 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598804 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598807 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598810 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598813 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598815 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598817 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598820 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598822 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598826 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598829 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598831 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598835 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598837 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598840 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598843 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598846 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598849 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598851 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598854 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.600061 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598857 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598860 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598862 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598865 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598868 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598871 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598873 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598876 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598879 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598881 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598884 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598886 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598889 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598892 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598897 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598902 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598906 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598909 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598912 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.600552 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598915 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598919 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598923 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598926 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598928 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598931 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598934 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598936 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598939 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598943 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598945 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598948 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598951 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598953 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598956 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598959 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598961 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598964 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598966 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.601047 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598969 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598972 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598974 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598977 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598980 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598983 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598985 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598988 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598992 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.598996 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599771 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599779 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599783 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599786 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599788 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599791 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599794 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599797 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599800 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599803 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.601517 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599806 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599808 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599811 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599813 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599816 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599818 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599821 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599824 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599827 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599829 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599832 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599835 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599838 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599841 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599844 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599846 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599849 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599851 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599854 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599856 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.602078 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599859 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599861 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599866 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599869 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599872 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599874 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599877 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599879 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599882 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599885 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599887 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599890 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599893 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599897 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599902 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599906 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599910 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599913 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599916 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.602617 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599919 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599923 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599926 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599928 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599931 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599933 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599937 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599941 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599944 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599947 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599950 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599952 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599955 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599957 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599960 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599962 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599966 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599969 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599971 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.603113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599974 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599977 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599979 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599982 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599984 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599987 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599989 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599992 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599994 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.599997 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600000 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600003 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600007 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600010 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600013 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600015 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600018 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600021 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600023 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600026 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.603602 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600029 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600031 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600034 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600036 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600039 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600041 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600044 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600046 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600049 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600052 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600055 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600057 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600060 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600063 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600066 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600068 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600072 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600075 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600589 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600594 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.604130 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600597 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600600 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600602 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600605 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600608 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600611 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600614 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600619 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600621 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600624 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600627 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600629 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600632 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600634 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600637 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600640 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600642 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600645 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600648 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600650 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.604635 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600653 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600656 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600665 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600668 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600671 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600674 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600677 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600680 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600682 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600685 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600688 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600690 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600693 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600710 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600713 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600716 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600718 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600721 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600724 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600728 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.605155 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600730 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600733 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600736 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600738 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600741 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600743 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600746 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600749 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600751 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600754 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600756 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600759 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600761 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600764 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600766 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600775 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600778 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600781 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600784 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.605651 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600787 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600789 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600792 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600795 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600797 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600800 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600802 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600805 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600808 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600811 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600813 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600816 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600821 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600825 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600829 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600831 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600834 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600837 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600840 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.606187 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600842 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600845 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600847 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600850 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600852 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600855 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600857 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600860 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600863 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600865 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600877 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600880 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600882 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600885 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600887 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600890 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600892 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600895 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600897 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600900 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.606654 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600902 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600905 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600908 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600910 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600915 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.600921 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601017 4948 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601024 4948 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601030 4948 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601035 4948 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601040 4948 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601043 4948 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601047 4948 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601052 4948 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601055 4948 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601058 4948 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601062 4948 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601066 4948 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601069 4948 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601072 4948 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601074 4948 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601078 4948 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601080 4948 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:37:50.607217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601090 4948 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601096 4948 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601099 4948 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601102 4948 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601105 4948 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601109 4948 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601113 4948 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601116 4948 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601119 4948 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601122 4948 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601125 4948 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601128 4948 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601131 4948 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601134 4948 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601139 4948 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601144 4948 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601147 4948 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601150 4948 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601153 4948 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601157 4948 flags.go:64] FLAG: --enable-server="true" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601160 4948 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601164 4948 flags.go:64] FLAG: --event-burst="100" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601167 4948 flags.go:64] FLAG: --event-qps="50" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601171 4948 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:37:50.608149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601174 4948 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601177 4948 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601182 4948 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601185 4948 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601188 4948 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601192 4948 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601195 4948 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601198 4948 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601201 4948 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601205 4948 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601208 4948 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601211 4948 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601214 4948 flags.go:64] FLAG: --feature-gates="" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601219 4948 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601222 4948 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601225 4948 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601228 4948 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601231 4948 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601234 4948 flags.go:64] FLAG: --help="false" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601237 4948 flags.go:64] FLAG: --hostname-override="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601241 4948 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601244 4948 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601248 4948 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:37:50.609232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601253 4948 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601257 4948 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601260 4948 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601263 4948 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601266 4948 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601269 4948 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601272 4948 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601275 4948 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601278 4948 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601281 4948 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601284 4948 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601286 4948 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601289 4948 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601292 4948 flags.go:64] FLAG: --lock-file="" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601295 4948 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601298 4948 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601301 4948 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601307 4948 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601310 4948 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601313 4948 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601316 4948 flags.go:64] FLAG: --logging-format="text" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601319 4948 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601323 4948 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601326 4948 flags.go:64] FLAG: --manifest-url="" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601329 4948 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:37:50.610324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601333 4948 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601336 4948 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601340 4948 flags.go:64] FLAG: --max-pods="110" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601343 4948 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601346 4948 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601349 4948 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601354 4948 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601359 4948 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601362 4948 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601365 4948 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601372 4948 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601376 4948 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601379 4948 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601382 4948 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601385 4948 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601388 4948 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601391 4948 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601394 4948 flags.go:64] FLAG: --port="10250" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601397 4948 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601400 4948 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0137315b5138f4b7d" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601404 4948 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601406 4948 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601410 4948 flags.go:64] FLAG: --register-node="true" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601413 4948 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601416 4948 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601419 4948 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:37:50.611491 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601422 4948 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601426 4948 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601430 4948 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601433 4948 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601436 4948 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601440 4948 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601443 4948 flags.go:64] FLAG: --runonce="false" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601446 4948 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601449 4948 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601452 4948 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601455 4948 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601458 4948 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601463 4948 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601469 4948 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601472 4948 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601475 4948 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601478 4948 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601481 4948 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601484 4948 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601487 4948 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601490 4948 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601493 4948 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601499 4948 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601502 4948 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601505 4948 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:37:50.612615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601512 4948 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601515 4948 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601518 4948 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601521 4948 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601525 4948 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601528 4948 flags.go:64] FLAG: --v="2" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601533 4948 flags.go:64] FLAG: --version="false" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601537 4948 flags.go:64] FLAG: --vmodule="" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601541 4948 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.601550 4948 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601744 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601749 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601753 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601756 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601759 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601762 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601764 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601767 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601772 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601779 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601786 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601790 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.613690 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601795 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601800 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601803 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601806 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601810 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601812 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601815 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601818 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601820 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601823 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601826 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601828 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601831 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601833 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601836 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601839 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601841 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601844 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601846 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601849 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.614665 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601859 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601862 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601865 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601867 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601870 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601873 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601875 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601879 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601883 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601888 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601895 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601898 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601901 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601904 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601907 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601909 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601912 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601915 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601917 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.615401 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601920 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601923 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601926 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601928 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601931 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601933 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601936 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601939 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601941 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601944 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601946 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601949 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601952 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601955 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601965 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601968 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601972 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601975 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601978 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601981 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.616010 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601983 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601986 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601989 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601994 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601997 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.601999 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602002 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602004 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602007 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602010 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602012 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602015 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602018 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602020 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602023 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602026 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602028 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602031 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602033 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602036 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.616712 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602039 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602041 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602044 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602046 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602049 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602051 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602054 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602062 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602065 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602068 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602071 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602073 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602076 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602078 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602082 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.602112 4948 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:50.617246 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602169 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602174 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602177 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602180 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602183 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602186 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602190 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602193 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602195 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602199 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602202 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602204 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602207 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602210 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602213 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602215 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602218 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602220 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602223 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.617643 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602226 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602228 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602231 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602233 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602236 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602245 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602247 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602250 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602253 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602256 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602258 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602263 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602265 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602269 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602272 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602275 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602278 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602280 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602283 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.618216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602285 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602288 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602291 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602293 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602296 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602298 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602301 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602304 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602306 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602309 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602311 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602314 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602316 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602319 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602322 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602325 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602327 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602330 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602333 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602341 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.618844 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602344 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602347 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602349 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602352 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602355 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602358 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602362 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602365 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602367 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602375 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602379 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602382 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602385 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602387 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602390 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602392 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602395 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602398 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602402 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.619480 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602406 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602410 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602413 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602417 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602420 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602422 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602425 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602428 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602430 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602433 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602436 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602439 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602441 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602449 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602452 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602455 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602458 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602462 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602465 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.619975 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602468 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602471 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602474 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602476 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602479 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602481 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602486 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602490 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602494 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.602505 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.606985 4948 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.607001 4948 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.607022 4948 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.607028 4948 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607930 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607942 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607948 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607958 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607963 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.620461 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607968 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607973 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607978 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607983 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607987 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607992 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.607997 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608002 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608006 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608011 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608016 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608028 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608032 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608037 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608042 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608047 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608051 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608056 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608060 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608065 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.620952 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608070 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608075 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608080 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608089 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608134 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608140 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608145 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608151 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608195 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608218 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608225 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608230 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608235 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608240 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608245 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608250 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608254 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608258 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608263 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608267 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.621440 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608271 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608282 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608286 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608291 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608310 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608327 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608332 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608336 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608341 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608346 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608349 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608352 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608355 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608383 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608527 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608534 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608541 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608546 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608552 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.621938 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608556 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608561 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608565 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608572 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608579 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608585 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608591 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608596 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608600 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608604 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608609 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608613 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608617 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608622 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608626 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608633 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608640 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608646 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608652 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.622407 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608657 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608662 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608666 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608670 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608674 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608684 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608690 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608695 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608719 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608723 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608728 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608733 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608737 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608742 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608746 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608750 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608755 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608759 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608764 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608768 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.623021 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608772 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608777 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.608781 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.608828 4948 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609075 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609083 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609088 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609094 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609099 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609104 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609110 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609114 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609118 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609123 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609127 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609131 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.623512 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609136 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609140 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609144 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609149 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609153 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609157 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609162 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609166 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609171 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609175 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609179 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609183 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609188 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609193 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609200 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609206 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609212 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609217 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609221 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.623911 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609226 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609230 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609234 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609238 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609242 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609247 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609251 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609256 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609260 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609264 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609269 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609273 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609277 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609282 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609286 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609290 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609295 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609299 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609303 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609307 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.624624 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609312 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609316 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609320 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609326 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609331 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609335 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609339 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609344 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609348 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609354 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609358 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609363 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609367 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609371 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609375 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609380 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609384 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609388 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.625115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609393 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609400 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609406 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609412 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609418 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609423 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609438 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609443 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609449 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609453 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609458 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609462 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609467 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609471 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609475 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609480 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609484 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609488 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609492 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.625576 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609496 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609501 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609506 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609510 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609515 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609519 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609523 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609527 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609532 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609536 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609541 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609545 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609549 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609554 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609559 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609563 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609567 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.626173 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609572 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.609610 4948 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609661 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609678 4948 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609683 4948 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609688 4948 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609692 4948 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609712 4948 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609717 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609721 4948 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609726 4948 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609730 4948 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609735 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609739 4948 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609744 4948 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609748 4948 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:50.626656 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609752 4948 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609756 4948 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609760 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609764 4948 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609768 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609772 4948 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609776 4948 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609779 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609783 4948 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609787 4948 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609791 4948 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609795 4948 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609799 4948 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609803 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609807 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609811 4948 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609816 4948 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609821 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609827 4948 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:50.627086 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609834 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609849 4948 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609854 4948 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609859 4948 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609863 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609867 4948 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609872 4948 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609876 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609880 4948 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609885 4948 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609889 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609893 4948 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609897 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609902 4948 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609906 4948 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609911 4948 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609915 4948 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609920 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609924 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609928 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:50.627834 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609933 4948 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609937 4948 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609941 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609945 4948 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609950 4948 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609954 4948 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609959 4948 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609963 4948 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609968 4948 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609972 4948 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609980 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609984 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609988 4948 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.609994 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610007 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610011 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610016 4948 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610020 4948 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610024 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:50.628538 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610028 4948 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610033 4948 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610037 4948 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610042 4948 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610049 4948 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610055 4948 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610060 4948 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610066 4948 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610070 4948 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610075 4948 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610079 4948 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610084 4948 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610089 4948 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610093 4948 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610097 4948 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610102 4948 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610107 4948 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610111 4948 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610115 4948 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:50.629255 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610119 4948 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610124 4948 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610128 4948 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610132 4948 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610139 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610143 4948 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610148 4948 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610152 4948 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610156 4948 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610168 4948 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610173 4948 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610180 4948 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610184 4948 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610188 4948 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:37:50.610192 4948 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.610273 4948 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.611824 4948 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.611902 4948 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.612177 4948 server.go:1014] "Starting client certificate rotation" Nov 03 18:37:50.629870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.612309 4948 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.612365 4948 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:32:08 +0000 UTC" deadline="2028-07-28 01:53:01.307332506 +0000 UTC" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.612392 4948 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15175h15m10.694942375s" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.612978 4948 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.614747 4948 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.616086 4948 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.617950 4948 log.go:25] "Validated CRI v1 image API" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.618772 4948 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:37:50.630350 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.620879 4948 fs.go:135] Filesystem UUIDs: map[13337f76-2d5d-414e-acff-5855b802586d:/dev/nvme0n1p4 6e91736e-d413-4447-96e6-0eb14e69c40a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 18:37:50.630636 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.620899 4948 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0387d9bbc097c437da6d05f4b2d13fe60e5df5490bc0d80f014cee1aaec41db3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0387d9bbc097c437da6d05f4b2d13fe60e5df5490bc0d80f014cee1aaec41db3/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/069dc0958b2c876b35b84e5ab0dc7d41c3d0e4c6dc4fc6000869ae7062d4af6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/069dc0958b2c876b35b84e5ab0dc7d41c3d0e4c6dc4fc6000869ae7062d4af6b/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08226d1e6a9a46d050f1d4ba28ad63976d864f51e1fd5094459bbeca87410150/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08226d1e6a9a46d050f1d4ba28ad63976d864f51e1fd5094459bbeca87410150/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19921ad23950c57880563672898b2bda149c771febc02109a741231fae0ba973/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19921ad23950c57880563672898b2bda149c771febc02109a741231fae0ba973/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/229e4f2c779762cb21e78cb31a9054c48f1d1bc11d4f4fe93c1f7826e9b3653f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/229e4f2c779762cb21e78cb31a9054c48f1d1bc11d4f4fe93c1f7826e9b3653f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24d36e323082a4a69a27793e13bb654eb1bb38322f97e064b4612f5549f7a05b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24d36e323082a4a69a27793e13bb654eb1bb38322f97e064b4612f5549f7a05b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b6d1b16d41588f5ed954db7693a192a86884bca7821cda6b90d55486e9a915a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b6d1b16d41588f5ed954db7693a192a86884bca7821cda6b90d55486e9a915a/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/406ee152fba5738bd51f49f57dc4923d333e334d790aa87bbf211050f82d1af0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/406ee152fba5738bd51f49f57dc4923d333e334d790aa87bbf211050f82d1af0/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4efec120778c48bafdef06b5ffd62e8b2f7ab4b39e4e85630938773f420a7d56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4efec120778c48bafdef06b5ffd62e8b2f7ab4b39e4e85630938773f420a7d56/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/616adb26588f4e73d754a9769c2169cdda6eb4671fe134cc8d42066bb18659f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/616adb26588f4e73d754a9769c2169cdda6eb4671fe134cc8d42066bb18659f9/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a3f01e433aeb94fed773b3d809c8a8c891d0f710f37cd1cc599433ce9e614b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a3f01e433aeb94fed773b3d809c8a8c891d0f710f37cd1cc599433ce9e614b1/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c368b178ed03df9d3eb3f49c050a2c621500cfab5b82170e6e433f7d2ceca8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c368b178ed03df9d3eb3f49c050a2c621500cfab5b82170e6e433f7d2ceca8c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e2dabf0642e933e3d4657c2df5e794f5b91e4d5300346d1fa08c6edf2bbb811/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e2dabf0642e933e3d4657c2df5e794f5b91e4d5300346d1fa08c6edf2bbb811/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/752ae2fa5ac98390145f5e9b63d8c6bc9f01c0ddac76235ca3a29aab204f2ad8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/752ae2fa5ac98390145f5e9b63d8c6bc9f01c0ddac76235ca3a29aab204f2ad8/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77f30296d3aaba6d9b86c4b006ba68536728dd86919b0df0f40d1e4f283392d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77f30296d3aaba6d9b86c4b006ba68536728dd86919b0df0f40d1e4f283392d8/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dc945b69a3be9076725fcd61450e852f1ab634ac66ed32f5d17d6a8ebbdc9ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dc945b69a3be9076725fcd61450e852f1ab634ac66ed32f5d17d6a8ebbdc9ef/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2acd7e013068985190ce35177b09ebb3a129c83c9755935aaf7ea05b2fb40af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2acd7e013068985190ce35177b09ebb3a129c83c9755935aaf7ea05b2fb40af/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3d9cde95c420b57810027daa9e2671d5d8c627f3d08c8ef86afbd9ddde693a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3d9cde95c420b57810027daa9e2671d5d8c627f3d08c8ef86afbd9ddde693a2/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3c460fd9e24abe8d65a62a828d63c0598577e9f7837b89e2e05416270602027/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3c460fd9e24abe8d65a62a828d63c0598577e9f7837b89e2e05416270602027/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6dc24380b0a21f21f405e06645803c376cc57765ed14707f317d94c2b5f0430/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6dc24380b0a21f21f405e06645803c376cc57765ed14707f317d94c2b5f0430/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f028eb3cb7a5ff0b6e9951ad48e5cd1d78dad1da208c47a5e7055b8836bf70cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f028eb3cb7a5ff0b6e9951ad48e5cd1d78dad1da208c47a5e7055b8836bf70cb/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/066c2936-7cba-4b26-8f35-ba18b055aef2/volumes/kubernetes.io~projected/kube-api-access-jv95g:{mountpoint:/var/lib/kubelet/pods/066c2936-7cba-4b26-8f35-ba18b055aef2/volumes/kubernetes.io~projected/kube-api-access-jv95g major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07926d6b-bf32-412b-89e5-26c0778a793f/volumes/kubernetes.io~projected/kube-api-access-75b52:{mountpoint:/var/lib/kubelet/pods/07926d6b-bf32-412b-89e5-26c0778a793f/volumes/kubernetes.io~projected/kube-api-access-75b52 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~projected/kube-api-access-rxbnd:{mountpoint:/var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~projected/kube-api-access-rxbnd major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~secret/default-certificate major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10497765-1e63-4159-870c-ede32ad1b170/volumes/kubernetes.io~projected/kube-api-access-br7cq:{mountpoint:/var/lib/kubelet/pods/10497765-1e63-4159-870c-ede32ad1b170/volumes/kubernetes.io~projected/kube-api-access-br7cq major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10497765-1e63-4159-870c-ede32ad1b170/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/10497765-1e63-4159-870c-ede32ad1b170/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~projected/kube-api-access-h8hdv:{mountpoint:/var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~projected/kube-api-access-h8hdv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d027ae4-4c9c-4beb-8134-3995b81d32c1/volumes/kubernetes.io~projected/kube-api-access-mffx2:{mountpoint:/var/lib/kubelet/pods/1d027ae4-4c9c-4beb-8134-3995b81d32c1/volumes/kubernetes.io~projected/kube-api-access-mffx2 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/245116c0-8b38-4fd5-868e-fe39362a2113/volumes/kubernetes.io~projected/kube-api-access-m76ms:{mountpoint:/var/lib/kubelet/pods/245116c0-8b38-4fd5-868e-fe39362a2113/volumes/kubernetes.io~projected/kube-api-access-m76ms major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/289c8acb-0083-4ec9-ab45-d5e94d47c490/volumes/kubernetes.io~projected/kube-api-access-ddmb4:{mountpoint:/var/lib/kubelet/pods/289c8acb-0083-4ec9-ab45-d5e94d47c490/volumes/kubernetes.io~projected/kube-api-access-ddmb4 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/289c8acb-0083-4ec9-ab45-d5e94d47c490/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/289c8acb-0083-4ec9-ab45-d5e94d47c490/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30383da3-bb10-420f-8f69-587bb4fdacd5/volumes/kubernetes.io~projected/kube-api-access-wfj5x:{mountpoint:/var/lib/kubelet/pods/30383da3-bb10-420f-8f69-587bb4fdacd5/volumes/kubernetes.io~projected/kube-api-access-wfj5x major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3441fcb6-fc61-446d-a986-79d01440c01b/volumes/kubernetes.io~projected/kube-api-access-7c8g8:{mountpoint:/var/lib/kubelet/pods/3441fcb6-fc61-446d-a986-79d01440c01b/volumes/kubernetes.io~projected/kube-api-access-7c8g8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a6b250a-51c5-4415-bd6b-4f1275517fdc/volumes/kubernetes.io~projected/kube-api-access-x64zv:{mountpoint:/var/lib/kubelet/pods/4a6b250a-51c5-4415-bd6b-4f1275517fdc/volumes/kubernetes.io~projected/kube-api-access-x64zv major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e6c6d42-58af-440c-8c07-915523a8a3bb/volumes/kubernetes.io~projected/kube-api-access-bcvpg:{mountpoint:/var/lib/kubelet/pods/4e6c6d42-58af-440c-8c07-915523a8a3bb/volumes/kubernetes.io~projected/kube-api-access-bcvpg major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5583debd-4dae-4fc4-8506-3a452b179306/volumes/kubernetes.io~projected/kube-api-access-jdbh4:{mountpoint:/var/lib/kubelet/pods/5583debd-4dae-4fc4-8506-3a452b179306/volumes/kubernetes.io~projected/kube-api-access-jdbh4 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5583debd-4dae-4fc4-8506-3a452b179306/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5583debd-4dae-4fc4-8506-3a452b179306/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6319a2b4-76e5-42dc-8097-a52570ed7f2c/volumes/kubernetes.io~projected/kube-api-access-2k6fd:{mountpoint:/var/lib/kubelet/pods/6319a2b4-76e5-42dc-8097-a52570ed7f2c/volumes/kubernetes.io~projected/kube-api-access-2k6fd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7d3c74-e08e-44b1-ab35-ced1363278f0/volumes/kubernetes.io~projected/kube-api-access-m5fj8:{mountpoint:/var/lib/kubelet/pods/6b7d3c74-e08e-44b1-ab35-ced1363278f0/volumes/kubernetes.io~projected/kube-api-access-m5fj8 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~projected/kube-api-access-2dzzp:{mountpoint:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~projected/kube-api-access-2dzzp major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f64f0eb-fed7-496b-92a6-84601b0bb544/volumes/kubernetes.io~projected/kube-api-access-8xg7h:{mountpoint:/var/lib/kubelet/pods/8f64f0eb-fed7-496b-92a6-84601b0bb544/volumes/kubernetes.io~projected/kube-api-access-8xg7h major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f64f0eb-fed7-496b-92a6-84601b0bb544/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8f64f0eb-fed7-496b-92a6-84601b0bb544/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a59c6f41-01de-40f1-9725-1ed9bd6784d6/volumes/kubernetes.io~projected/kube-api-access-gjs9k:{mountpoint:/var/lib/kubelet/pods/a59c6f41-01de-40f1-9725-1ed9bd6784d6/volumes/kubernetes.io~projected/kube-api-access-gjs9k major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af92784a-e931-457d-a327-5fbba2f37af5/volumes/kubernetes.io~projected/kube-api-access-sp7lf:{mountpoint:/var/lib/kubelet/pods/af92784a-e931-457d-a327-5fbba2f37af5/volumes/kubernetes.io~projected/kube-api-access-sp7lf major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3e13ce6-21d8-4d9c-b589-46699c28b856/volumes/kubernetes.io~projected/kube-api-access-gjg9j:{mountpoint:/var/lib/kubelet/pods/c3e13ce6-21d8-4d9c-b589-46699c28b856/volumes/kubernetes.io~projected/kube-api-access-gjg9j major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volumes/kubernetes.io~projected/kube-api-access-pmrnd:{mountpoint:/var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volumes/kubernetes.io~projected/kube-api-access-pmrnd major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ee858f-ae52-4de9-b576-ed63754072f9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d0ee858f-ae52-4de9-b576-ed63754072f9/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f084d2-6540-44ee-ac2b-c8e6df92a050/volumes/kubernetes.io~projected/kube-api-access-mb5nn:{mountpoint:/var/lib/kubelet/pods/d0f084d2-6540-44ee-ac2b-c8e6df92a050/volumes/kubernetes.io~projected/kube-api-access-mb5nn major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~projected/kube-api-access-7vqbv:{mountpoint:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~projected/kube-api-access-7vqbv major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4c6be41-6db5-4be6-8cac-76b00171f6af/volumes/kubernetes.io~projected/kube-api-access-2qlpc:{mountpoint:/var/lib/kubelet/pods/d4c6be41-6db5-4be6-8cac-76b00171f6af/volumes/kubernetes.io~projected/kube-api-access-2qlpc major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e50d63ba-d363-48f5-a720-880cd7c895d0/volumes/kubernetes.io~projected/kube-api-access-kkn57:{mountpoint:/var/lib/kubelet/pods/e50d63ba-d363-48f5-a720-880cd7c895d0/volumes/kubernetes.io~projected/kube-api-access-kkn57 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e50d63ba-d363-48f5-a720-880cd7c895d0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e50d63ba-d363-48f5-a720-880cd7c895d0/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edef417a-da49-4ca8-a744-0a3c7569546f/volumes/kubernetes.io~projected/kube-api-access-7b8cj:{mountpoint:/var/lib/kubelet/pods/edef417a-da49-4ca8-a744-0a3c7569546f/volumes/kubernetes.io~projected/kube-api-access-7b8cj major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f/volumes/kubernetes.io~projected/kube-api-access-c64s6:{mountpoint:/var/lib/kubelet/pods/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f/volumes/kubernetes.io~projected/kube-api-access-c64s6 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe291b57-902f-430e-91ed-d373c931c885/volumes/kubernetes.io~projected/kube-api-access-2jqqt:{mountpoint:/var/lib/kubelet/pods/fe291b57-902f-430e-91ed-d373c931c885/volumes/kubernetes.io~projected/kube-api-access-2jqqt major:0 minor:231 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/38135ad77976e3cf70cc9bf9bc1108f467844582cf8f3fcb68f4a7e349590930/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/217336f9f9202ad88d4d2916fa5e9ceb2006a969c5fa6039fc8386de03fa8da5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6e9823519251afb89c9118dec0b6315ce27d482da465f1bc080f8d9073d96f02/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2dcd145f7e78b0a74d8b7dc2897392963850068c4b30c046cebbe9c2115691a9/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/052e8cb3161daa8d03127a7f4d058740c260312bfafb7223f25634efe5649aa4/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a8115a1f24f08812407561d1487784fa97fb7bb87aa6c85986c67d007b593ca3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/500e944f49a0990444f26260bd975b970ce5b9a20ca43a0691c9344cc6247588/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ff77a4cc9180f653dd4b81c8967c1160b7951eba3dc08bb6a8f53a1d695c876f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/9dedcbb307d1093ea06bdfe082f863b27cc50f3598846819d988543e05f49333/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/dd47cb518494a65c7613a9c01b756a3478f9ac5ebb00d7655b0feeec3ad219ba/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/640ecd45c0e008f8b2a7caa2ef0bc1286187ca947f80add90efbf6253cb1d0a5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/775f4123a2051816aa539393071ab9be83f5b11d30c16ce35b587a43a3efed8b/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e9e0504550ecc9b927aa9390a42fcc69a4226e19ce7ab64a9a7a5c82bf5c944f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/27565f4bc3d6a1440f10c63d85b56db2afde104479069eec2c5c46eef4ebaf0d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ab4244b3655ecdfbfc9513464307e133d1b66bc90af68f47640c4612fcf51fbe/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3be0a356e774a5b7f57f29cb87ac38de91294ead78cd0578cb1b9dc1db37bb3e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b70a2b9449a2d4f69682ec19c4ed8171ecd8eabb07afe070336bd8c4ec282995/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cd80a99e5d31c1d25d8be09e548c2f2ac09cefa7226c0faadee6850013a2e2a8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/474d417d7a15af34d5e5447257dfa697bb2a781b0b22a71a61f6ee3d36fce69b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/edfa9d2d9bacd50cfabcea8eb6eca791f089e241433a42031b928baa3d65f83b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/436394b84dd4599a050de49a25523f19611d0a5b42e917ff1b62e647f63e039a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ca6cfbcb7c354cb4309f8441819a1ecb91cdb6c74d6e261405868f8464ab8152/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/4a229c1b22cb3c15df62b6c4bcf0719db9786925fdec8dc9f3266957a60030b2/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/ce78da6537b840e49b4fb1b9d69d31e81dc90798ddad3404aaf51f5559701139/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/37afa9246790e2e180010181d3ee0e4cfd8acc41c2d040b25728915e44077a1f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1004b0030e2ca62b14d9a5971cac4dccc0aa88db2130e9f23e09c94f7ff795eb/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/729a3805a4125f231943b093a838056676eeb75694fc698e47ad0f16e32b0df4/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/5eb387eef78449db031b1dd90627571f9284f22de0a48b44896b93ed6eccceda/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/b14996c7230adad38f4394118d08169558a15df83f70f739da09b49924d27836/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/8849ce93261a92a90c5f5c7f8c3b773ba9ce2e99255d35773f0f60b40d30c0b6/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/1bd2c21d6ae64e3f9c1f754088a4041dcfe5e525d2073df8eb1c3ece4ad5b415/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/32a0d65275af60d4593af69f125a808d32a5362310aa1aa6152572c1763258bb/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/ee9e1e795206bad835715acba3f04f74ca86179ac415a164d9dffe9866546ace/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/42deac82f4df55a1a9abf4f465ff37866926435bdb0377f76a0006cacbc27245/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/5d492c29769ffd457c3a829600449b3ed7345e1e2c0ec1c25029e12bee49ce30/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/facf1ff250fd0a01ec7304d0ab65be851a0ff285588b1ed42da2fe5b606455a3/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/347b32751611cbd0a8fc32a54a28d6fe9b58bf79a635e195cc6a8ae0670779df/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/8019c5871c524a9a52d113529b975d32bbd008dfa3cd2cc4e3005839fe70b798/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6ae9ebbcfaaf62ac026e8e49cb85c410e44494bfe087e49984b713633838baf6/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/bacf4cff9176476f8bc58237f18bef6b40988ca4db4d40d94b12d1a7198f5f9c/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/63836e118c452354e95369b517540801977a5fd537be3e51a050721845a9c198/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6554e6f380c64f7078fcf1115ac1aa22d6cb961dca1dc5f9069ff87a89ce98aa/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/474f42fdf40561436d0b8b40f6fa2d6b6d88b05616ef8b9f13088ac58694c360/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/578a2572e8cb238259eeed62fe6e9050d8c83920a35a4d3f802c4446e5fe27e8/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/36f6560cfeaaf98636d0b9fc51f9a291f122c348ad35c3e04e374a7a90632637/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/d496b348ec391d08f60ccdd5819a1a807805bfe96d574c8139edc65e4edcfa5b/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/3aebfc96d08a130e734003625450f1929cd9576eefeb39e6cade72d20d7b6b6d/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/62c8c92be85c681763363cfb3744c8f6d8561464198f601109c74ba4747316b1/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6607f62fedf462d71d25609ddaed47ed2c5a06d48439f94beb446673f4fc46ea/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6291d26ac99b60d4e2b584c8f7ae730a23ab815c6139e3cb2ff51ba17e5aa9ec/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/fc1498b11933549d454a82cdf972ff96fecf4d8402aaddcfdeb90bc9e977bc59/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7589cb7b33210a8ef5c3449e7032fa1ada39eb6dd8c4a08ce4396990ad06271f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/b6f1c63dd2b5ce30816453ec440166e90fdba2610f90aa7bb8db7f077bc0b024/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fcd1dc95c9b38bc619d72cd4c81ce2f374f845987e29b147bfc944d3d7a13762/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/68ff2a4cf4db447d6fff6f2045e169bd073475d4a9349a1ad666915d187a71d6/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 18:37:50.630882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.630421 4948 manager.go:217] Machine: {Timestamp:2026-11-03 18:37:50.629945961 +0000 UTC m=+0.112941866 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096784 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec253c03869f63ed55da7a07b82e13c9 SystemUUID:ec253c03-869f-63ed-55da-7a07b82e13c9 BootID:4e52d022-1f13-48cc-86e9-ac5a819e27f3 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d0ee858f-ae52-4de9-b576-ed63754072f9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/245116c0-8b38-4fd5-868e-fe39362a2113/volumes/kubernetes.io~projected/kube-api-access-m76ms DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3c460fd9e24abe8d65a62a828d63c0598577e9f7837b89e2e05416270602027/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b6d1b16d41588f5ed954db7693a192a86884bca7821cda6b90d55486e9a915a/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e2dabf0642e933e3d4657c2df5e794f5b91e4d5300346d1fa08c6edf2bbb811/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3441fcb6-fc61-446d-a986-79d01440c01b/volumes/kubernetes.io~projected/kube-api-access-7c8g8 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f64f0eb-fed7-496b-92a6-84601b0bb544/volumes/kubernetes.io~projected/kube-api-access-8xg7h DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dc945b69a3be9076725fcd61450e852f1ab634ac66ed32f5d17d6a8ebbdc9ef/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a6b250a-51c5-4415-bd6b-4f1275517fdc/volumes/kubernetes.io~projected/kube-api-access-x64zv DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5583debd-4dae-4fc4-8506-3a452b179306/volumes/kubernetes.io~projected/kube-api-access-jdbh4 DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/616adb26588f4e73d754a9769c2169cdda6eb4671fe134cc8d42066bb18659f9/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19921ad23950c57880563672898b2bda149c771febc02109a741231fae0ba973/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e50d63ba-d363-48f5-a720-880cd7c895d0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a3f01e433aeb94fed773b3d809c8a8c891d0f710f37cd1cc599433ce9e614b1/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0387d9bbc097c437da6d05f4b2d13fe60e5df5490bc0d80f014cee1aaec41db3/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4c6be41-6db5-4be6-8cac-76b00171f6af/volumes/kubernetes.io~projected/kube-api-access-2qlpc DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/edef417a-da49-4ca8-a744-0a3c7569546f/volumes/kubernetes.io~projected/kube-api-access-7b8cj DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30383da3-bb10-420f-8f69-587bb4fdacd5/volumes/kubernetes.io~projected/kube-api-access-wfj5x DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a59c6f41-01de-40f1-9725-1ed9bd6784d6/volumes/kubernetes.io~projected/kube-api-access-gjs9k DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/229e4f2c779762cb21e78cb31a9054c48f1d1bc11d4f4fe93c1f7826e9b3653f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af92784a-e931-457d-a327-5fbba2f37af5/volumes/kubernetes.io~projected/kube-api-access-sp7lf DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f028eb3cb7a5ff0b6e9951ad48e5cd1d78dad1da208c47a5e7055b8836bf70cb/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5583debd-4dae-4fc4-8506-3a452b179306/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08226d1e6a9a46d050f1d4ba28ad63976d864f51e1fd5094459bbeca87410150/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/07926d6b-bf32-412b-89e5-26c0778a793f/volumes/kubernetes.io~projected/kube-api-access-75b52 DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6dc24380b0a21f21f405e06645803c376cc57765ed14707f317d94c2b5f0430/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~projected/kube-api-access-rxbnd DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:267 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/406ee152fba5738bd51f49f57dc4923d333e334d790aa87bbf211050f82d1af0/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/066c2936-7cba-4b26-8f35-ba18b055aef2/volumes/kubernetes.io~projected/kube-api-access-jv95g DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~projected/kube-api-access-7vqbv DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3d9cde95c420b57810027daa9e2671d5d8c627f3d08c8ef86afbd9ddde693a2/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e50d63ba-d363-48f5-a720-880cd7c895d0/volumes/kubernetes.io~projected/kube-api-access-kkn57 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe291b57-902f-430e-91ed-d373c931c885/volumes/kubernetes.io~projected/kube-api-access-2jqqt DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f64f0eb-fed7-496b-92a6-84601b0bb544/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/069dc0958b2c876b35b84e5ab0dc7d41c3d0e4c6dc4fc6000869ae7062d4af6b/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e6c6d42-58af-440c-8c07-915523a8a3bb/volumes/kubernetes.io~projected/kube-api-access-bcvpg DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/289c8acb-0083-4ec9-ab45-d5e94d47c490/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d027ae4-4c9c-4beb-8134-3995b81d32c1/volumes/kubernetes.io~projected/kube-api-access-mffx2 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24d36e323082a4a69a27793e13bb654eb1bb38322f97e064b4612f5549f7a05b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6319a2b4-76e5-42dc-8097-a52570ed7f2c/volumes/kubernetes.io~projected/kube-api-access-2k6fd DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/752ae2fa5ac98390145f5e9b63d8c6bc9f01c0ddac76235ca3a29aab204f2ad8/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c368b178ed03df9d3eb3f49c050a2c621500cfab5b82170e6e433f7d2ceca8c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f/volumes/kubernetes.io~projected/kube-api-access-c64s6 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10497765-1e63-4159-870c-ede32ad1b170/volumes/kubernetes.io~projected/kube-api-access-br7cq DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77f30296d3aaba6d9b86c4b006ba68536728dd86919b0df0f40d1e4f283392d8/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volumes/kubernetes.io~projected/kube-api-access-pmrnd DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/289c8acb-0083-4ec9-ab45-d5e94d47c490/volumes/kubernetes.io~projected/kube-api-access-ddmb4 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4efec120778c48bafdef06b5ffd62e8b2f7ab4b39e4e85630938773f420a7d56/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7d3c74-e08e-44b1-ab35-ced1363278f0/volumes/kubernetes.io~projected/kube-api-access-m5fj8 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2acd7e013068985190ce35177b09ebb3a129c83c9755935aaf7ea05b2fb40af/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10497765-1e63-4159-870c-ede32ad1b170/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c3e13ce6-21d8-4d9c-b589-46699c28b856/volumes/kubernetes.io~projected/kube-api-access-gjg9j DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f084d2-6540-44ee-ac2b-c8e6df92a050/volumes/kubernetes.io~projected/kube-api-access-mb5nn DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/175d5971-984f-4e61-bdaa-f1b34609f0e4/volumes/kubernetes.io~projected/kube-api-access-h8hdv DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes/kubernetes.io~projected/kube-api-access-2dzzp DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce2ce282-8a19-40f3-a97f-a270daedbe66/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0387d9bbc097c43 MacAddress:f6:24:95:a2:be:de Speed:10000 Mtu:8901} {Name:069dc0958b2c876 MacAddress:06:16:97:8d:53:22 Speed:10000 Mtu:8901} {Name:08226d1e6a9a46d MacAddress:ce:44:3b:1c:de:a9 Speed:10000 Mtu:8901} {Name:19921ad23950c57 MacAddress:2e:ec:f4:fe:f5:4b Speed:10000 Mtu:8901} {Name:229e4f2c779762c MacAddress:8a:d8:96:68:3a:ff Speed:10000 Mtu:8901} {Name:3b6d1b16d41588f MacAddress:ee:2b:1e:4c:66:a7 Speed:10000 Mtu:8901} {Name:406ee152fba5738 MacAddress:0e:4c:cb:94:45:0b Speed:10000 Mtu:8901} {Name:4efec120778c48b MacAddress:ee:86:71:88:c1:d5 Speed:10000 Mtu:8901} {Name:6e2dabf0642e933 MacAddress:ba:f9:64:e6:5a:62 Speed:10000 Mtu:8901} {Name:752ae2fa5ac9839 MacAddress:c6:97:86:67:d0:24 Speed:10000 Mtu:8901} {Name:77f30296d3aaba6 MacAddress:e2:9a:4a:2d:18:0b Speed:10000 Mtu:8901} {Name:9dc945b69a3be90 MacAddress:da:64:b7:b0:7a:f6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c4:de:0a:a1 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:7c:a5:e3:65:42 Speed:0 Mtu:8901} {Name:c3c460fd9e24abe MacAddress:0a:b2:f1:39:50:b6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c4:de:0a:a1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d6:48:2d:47:59:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:85:1f:fe:64:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:37:50.630882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.630783 4948 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:37:50.630882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.630830 4948 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:37:50.631115 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631082 4948 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:37:50.631225 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631101 4948 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-95.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:37:50.631266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631226 4948 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:37:50.631266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631235 4948 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:37:50.631266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631248 4948 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:37:50.631344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631268 4948 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:37:50.631344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631287 4948 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:37:50.631442 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631433 4948 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:37:50.631538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631529 4948 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:37:50.631574 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631565 4948 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:37:50.631574 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631573 4948 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:37:50.631640 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631583 4948 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:37:50.631640 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631592 4948 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:37:50.631640 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.631600 4948 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:37:50.632303 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.632287 4948 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:37:50.632303 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.632301 4948 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:37:50.632775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.632759 4948 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:37:50.701087 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701066 4948 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:37:50.701087 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701084 4948 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:37:50.701236 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701225 4948 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 18:37:50.701507 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701494 4948 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:37:50.701538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701526 4948 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:37:50.701664 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701655 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:37:50.701711 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701672 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:37:50.701711 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701679 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:37:50.701711 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701687 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:37:50.701711 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701710 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701721 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701729 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701735 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701742 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701749 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701766 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701775 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701804 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:37:50.701820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.701810 4948 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:37:50.705558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.705542 4948 server.go:1263] "Started kubelet" Nov 03 18:37:50.706115 ip-10-0-128-95 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:37:50.706270 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.706104 4948 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:37:50.706431 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.706343 4948 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:37:50.706851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.706433 4948 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:37:50.706851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.706770 4948 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:37:50.708831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.707838 4948 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:37:50.711511 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.711207 4948 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:37:50.712871 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.712849 4948 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:37:50.713852 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.713828 4948 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" Nov 03 18:37:50.714094 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.714074 4948 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:37:50.714094 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.714065 4948 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:09 +0000 UTC" deadline="2028-05-15 00:11:26.730537051 +0000 UTC" Nov 03 18:37:50.714094 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.714095 4948 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:37:50.714283 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.714100 4948 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13397h33m36.016440686s" Nov 03 18:37:50.714283 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.714116 4948 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:37:50.717489 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717468 4948 factory.go:55] Registering systemd factory Nov 03 18:37:50.717587 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717495 4948 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:37:50.717900 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717883 4948 factory.go:153] Registering CRI-O factory Nov 03 18:37:50.718000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717904 4948 factory.go:223] Registration of the crio container factory successfully Nov 03 18:37:50.718000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717909 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" volumeName="kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token" seLinuxMountContext="" Nov 03 18:37:50.718000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717955 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10497765-1e63-4159-870c-ede32ad1b170" volumeName="kubernetes.io/projected/10497765-1e63-4159-870c-ede32ad1b170-kube-api-access-br7cq" seLinuxMountContext="" Nov 03 18:37:50.718000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717973 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" volumeName="kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration" seLinuxMountContext="" Nov 03 18:37:50.718000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717981 4948 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:37:50.718000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.717987 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a59c6f41-01de-40f1-9725-1ed9bd6784d6" volumeName="kubernetes.io/projected/a59c6f41-01de-40f1-9725-1ed9bd6784d6-kube-api-access-gjs9k" seLinuxMountContext="" Nov 03 18:37:50.718000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718000 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ee858f-ae52-4de9-b576-ed63754072f9" volumeName="kubernetes.io/configmap/d0ee858f-ae52-4de9-b576-ed63754072f9-konnectivity-ca" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718013 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" volumeName="kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718025 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50d63ba-d363-48f5-a720-880cd7c895d0" volumeName="kubernetes.io/configmap/e50d63ba-d363-48f5-a720-880cd7c895d0-config" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718037 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd5c02ff-a13f-4d45-bb84-df74ec8fd86f" volumeName="kubernetes.io/configmap/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-telemetry-config" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718053 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10497765-1e63-4159-870c-ede32ad1b170" volumeName="kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-tmp" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718066 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d027ae4-4c9c-4beb-8134-3995b81d32c1" volumeName="kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cni-binary-copy" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718082 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce2ce282-8a19-40f3-a97f-a270daedbe66" volumeName="kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-script-lib" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718004 4948 factory.go:103] Registering Raw factory Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718094 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" volumeName="kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718116 4948 manager.go:1199] Started watching for new ooms in manager Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718118 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50d63ba-d363-48f5-a720-880cd7c895d0" volumeName="kubernetes.io/projected/e50d63ba-d363-48f5-a720-880cd7c895d0-kube-api-access-kkn57" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718129 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd5c02ff-a13f-4d45-bb84-df74ec8fd86f" volumeName="kubernetes.io/projected/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-kube-api-access-c64s6" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718140 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d027ae4-4c9c-4beb-8134-3995b81d32c1" volumeName="kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-daemon-config" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718148 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f084d2-6540-44ee-ac2b-c8e6df92a050" volumeName="kubernetes.io/projected/d0f084d2-6540-44ee-ac2b-c8e6df92a050-kube-api-access-mb5nn" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718160 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe291b57-902f-430e-91ed-d373c931c885" volumeName="kubernetes.io/projected/fe291b57-902f-430e-91ed-d373c931c885-kube-api-access-2jqqt" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718173 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10497765-1e63-4159-870c-ede32ad1b170" volumeName="kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-service-ca-bundle" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718182 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3441fcb6-fc61-446d-a986-79d01440c01b" volumeName="kubernetes.io/configmap/3441fcb6-fc61-446d-a986-79d01440c01b-serviceca" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718191 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6319a2b4-76e5-42dc-8097-a52570ed7f2c" volumeName="kubernetes.io/empty-dir/6319a2b4-76e5-42dc-8097-a52570ed7f2c-tmp-dir" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718202 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7d3c74-e08e-44b1-ab35-ced1363278f0" volumeName="kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718211 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" volumeName="kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718218 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f64f0eb-fed7-496b-92a6-84601b0bb544" volumeName="kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-config" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718226 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce2ce282-8a19-40f3-a97f-a270daedbe66" volumeName="kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-config" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718234 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="175d5971-984f-4e61-bdaa-f1b34609f0e4" volumeName="kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-tmp" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718241 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3441fcb6-fc61-446d-a986-79d01440c01b" volumeName="kubernetes.io/projected/3441fcb6-fc61-446d-a986-79d01440c01b-kube-api-access-7c8g8" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718249 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5583debd-4dae-4fc4-8506-3a452b179306" volumeName="kubernetes.io/projected/5583debd-4dae-4fc4-8506-3a452b179306-kube-api-access-jdbh4" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718257 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" volumeName="kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718287 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c6be41-6db5-4be6-8cac-76b00171f6af" volumeName="kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-binary-copy" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718305 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245116c0-8b38-4fd5-868e-fe39362a2113" volumeName="kubernetes.io/projected/245116c0-8b38-4fd5-868e-fe39362a2113-kube-api-access-m76ms" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718315 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6319a2b4-76e5-42dc-8097-a52570ed7f2c" volumeName="kubernetes.io/projected/6319a2b4-76e5-42dc-8097-a52570ed7f2c-kube-api-access-2k6fd" seLinuxMountContext="" Nov 03 18:37:50.718310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718325 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a59c6f41-01de-40f1-9725-1ed9bd6784d6" volumeName="kubernetes.io/configmap/a59c6f41-01de-40f1-9725-1ed9bd6784d6-iptables-alerter-script" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718345 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e13ce6-21d8-4d9c-b589-46699c28b856" volumeName="kubernetes.io/projected/c3e13ce6-21d8-4d9c-b589-46699c28b856-kube-api-access-gjg9j" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718356 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" volumeName="kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718378 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" volumeName="kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718387 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30383da3-bb10-420f-8f69-587bb4fdacd5" volumeName="kubernetes.io/configmap/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-config" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718397 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e6c6d42-58af-440c-8c07-915523a8a3bb" volumeName="kubernetes.io/projected/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-api-access-bcvpg" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718405 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" volumeName="kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718423 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9" volumeName="kubernetes.io/projected/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-kube-api-access-rxbnd" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718434 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d027ae4-4c9c-4beb-8134-3995b81d32c1" volumeName="kubernetes.io/projected/1d027ae4-4c9c-4beb-8134-3995b81d32c1-kube-api-access-mffx2" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718443 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="289c8acb-0083-4ec9-ab45-d5e94d47c490" volumeName="kubernetes.io/configmap/289c8acb-0083-4ec9-ab45-d5e94d47c490-config" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718450 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" volumeName="kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718465 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce2ce282-8a19-40f3-a97f-a270daedbe66" volumeName="kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-env-overrides" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718474 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c6be41-6db5-4be6-8cac-76b00171f6af" volumeName="kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718484 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e6c6d42-58af-440c-8c07-915523a8a3bb" volumeName="kubernetes.io/empty-dir/4e6c6d42-58af-440c-8c07-915523a8a3bb-data-volume" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718494 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8da56f0f-6ab9-4eb3-ad1d-35ce98845ada" volumeName="kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718512 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c6be41-6db5-4be6-8cac-76b00171f6af" volumeName="kubernetes.io/projected/d4c6be41-6db5-4be6-8cac-76b00171f6af-kube-api-access-2qlpc" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718522 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edef417a-da49-4ca8-a744-0a3c7569546f" volumeName="kubernetes.io/projected/edef417a-da49-4ca8-a744-0a3c7569546f-kube-api-access-7b8cj" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718531 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9" volumeName="kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-default-certificate" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718541 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="175d5971-984f-4e61-bdaa-f1b34609f0e4" volumeName="kubernetes.io/projected/175d5971-984f-4e61-bdaa-f1b34609f0e4-kube-api-access-h8hdv" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718551 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="289c8acb-0083-4ec9-ab45-d5e94d47c490" volumeName="kubernetes.io/projected/289c8acb-0083-4ec9-ab45-d5e94d47c490-kube-api-access-ddmb4" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718559 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" volumeName="kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718569 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edef417a-da49-4ca8-a744-0a3c7569546f" volumeName="kubernetes.io/empty-dir/edef417a-da49-4ca8-a744-0a3c7569546f-tmp-dir" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718580 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="066c2936-7cba-4b26-8f35-ba18b055aef2" volumeName="kubernetes.io/projected/066c2936-7cba-4b26-8f35-ba18b055aef2-kube-api-access-jv95g" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718590 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9" volumeName="kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-stats-auth" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718602 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10497765-1e63-4159-870c-ede32ad1b170" volumeName="kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-snapshots" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718605 4948 manager.go:319] Starting recovery of all containers Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718619 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" volumeName="kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718628 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce2ce282-8a19-40f3-a97f-a270daedbe66" volumeName="kubernetes.io/projected/ce2ce282-8a19-40f3-a97f-a270daedbe66-kube-api-access-pmrnd" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718637 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" volumeName="kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718645 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07926d6b-bf32-412b-89e5-26c0778a793f" volumeName="kubernetes.io/projected/07926d6b-bf32-412b-89e5-26c0778a793f-kube-api-access-75b52" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718655 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="175d5971-984f-4e61-bdaa-f1b34609f0e4" volumeName="kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-tuned" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718664 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30383da3-bb10-420f-8f69-587bb4fdacd5" volumeName="kubernetes.io/projected/30383da3-bb10-420f-8f69-587bb4fdacd5-kube-api-access-wfj5x" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718675 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a6b250a-51c5-4415-bd6b-4f1275517fdc" volumeName="kubernetes.io/projected/4a6b250a-51c5-4415-bd6b-4f1275517fdc-kube-api-access-x64zv" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718684 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e6c6d42-58af-440c-8c07-915523a8a3bb" volumeName="kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718693 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f64f0eb-fed7-496b-92a6-84601b0bb544" volumeName="kubernetes.io/secret/8f64f0eb-fed7-496b-92a6-84601b0bb544-serving-cert" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718720 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f023639-2351-4534-8884-8ffdc05a4144" volumeName="kubernetes.io/configmap/0f023639-2351-4534-8884-8ffdc05a4144-nginx-conf" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718730 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5583debd-4dae-4fc4-8506-3a452b179306" volumeName="kubernetes.io/secret/5583debd-4dae-4fc4-8506-3a452b179306-cert" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718739 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f64f0eb-fed7-496b-92a6-84601b0bb544" volumeName="kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-trusted-ca" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718752 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edef417a-da49-4ca8-a744-0a3c7569546f" volumeName="kubernetes.io/configmap/edef417a-da49-4ca8-a744-0a3c7569546f-config-volume" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718761 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10497765-1e63-4159-870c-ede32ad1b170" volumeName="kubernetes.io/secret/10497765-1e63-4159-870c-ede32ad1b170-serving-cert" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718770 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f64f0eb-fed7-496b-92a6-84601b0bb544" volumeName="kubernetes.io/projected/8f64f0eb-fed7-496b-92a6-84601b0bb544-kube-api-access-8xg7h" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718780 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ee858f-ae52-4de9-b576-ed63754072f9" volumeName="kubernetes.io/secret/d0ee858f-ae52-4de9-b576-ed63754072f9-agent-certs" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718789 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c6be41-6db5-4be6-8cac-76b00171f6af" volumeName="kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718800 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50d63ba-d363-48f5-a720-880cd7c895d0" volumeName="kubernetes.io/secret/e50d63ba-d363-48f5-a720-880cd7c895d0-serving-cert" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718810 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10497765-1e63-4159-870c-ede32ad1b170" volumeName="kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-trusted-ca-bundle" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718822 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="289c8acb-0083-4ec9-ab45-d5e94d47c490" volumeName="kubernetes.io/secret/289c8acb-0083-4ec9-ab45-d5e94d47c490-serving-cert" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718832 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" volumeName="kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718841 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af92784a-e931-457d-a327-5fbba2f37af5" volumeName="kubernetes.io/projected/af92784a-e931-457d-a327-5fbba2f37af5-kube-api-access-sp7lf" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718851 4948 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce2ce282-8a19-40f3-a97f-a270daedbe66" volumeName="kubernetes.io/secret/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718860 4948 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:37:50.719785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.718868 4948 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:37:50.724060 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.723991 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/1.log" Nov 03 18:37:50.724621 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.724573 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/1.log" Nov 03 18:37:50.734831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.734806 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:37:50.734831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.734820 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:37:50.743918 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.743886 4948 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:37:50.745242 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.745219 4948 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:37:50.745242 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.745242 4948 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:37:50.745400 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.745263 4948 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:37:50.745400 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.745304 4948 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 18:37:50.771827 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.771801 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:37:50.772184 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.772159 4948 generic.go:356] "Generic (PLEG): container finished" podID="ce2ce282-8a19-40f3-a97f-a270daedbe66" containerID="94e7e3976e59ce02aae9240f907db86edc530047d91bbf6ef3159f19e919893d" exitCode=1 Nov 03 18:37:50.779751 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.779725 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/1.log" Nov 03 18:37:50.779840 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.779765 4948 generic.go:356] "Generic (PLEG): container finished" podID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerID="9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb" exitCode=255 Nov 03 18:37:50.792864 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.792836 4948 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="c9606ebfe68cc38d35b2f69baab44d6bbd803a6f2491baa4b2f55de824e5d23f" exitCode=0 Nov 03 18:37:50.792864 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.792860 4948 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="94050adafc46af6a6f7a2e54358f7121e9e95c85fab39a582484a7d86a884eb6" exitCode=0 Nov 03 18:37:50.792864 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.792867 4948 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="253498e11436a9a3941dff8315793a62f4a9bb5c212924efa2d7de5676a32bf8" exitCode=0 Nov 03 18:37:50.793096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.792873 4948 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="45b0f9931a0d84621b69f10b2e14455b73908241e7ac67707ae94ed7feaf578a" exitCode=0 Nov 03 18:37:50.793096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.792879 4948 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="0e587888ed4eb7668678eaf773cb284c0f496691dff7977abf868c03f9e695e4" exitCode=0 Nov 03 18:37:50.793096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.792884 4948 generic.go:356] "Generic (PLEG): container finished" podID="d4c6be41-6db5-4be6-8cac-76b00171f6af" containerID="c7fb616a25b28d842d00541ac59e578cced126b12fcb4ee19aef37a0623e9eef" exitCode=0 Nov 03 18:37:50.796894 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.796861 4948 generic.go:356] "Generic (PLEG): container finished" podID="8a249de0ac9079d1551a521e9f3cd5d3" containerID="6e559dd37ee6a38b4c4a73dbadc1b90e667a6b08e0bacc4f17163e4dc4882d59" exitCode=0 Nov 03 18:37:50.809949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.809933 4948 manager.go:324] Recovery completed Nov 03 18:37:50.814681 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.814660 4948 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" Nov 03 18:37:50.828457 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.828428 4948 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:50.829794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.829733 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:50.829794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.829761 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:50.829794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.829774 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:50.833124 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833108 4948 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:37:50.833124 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833123 4948 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:37:50.833216 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833126 4948 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:50.833216 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833143 4948 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:37:50.833324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833296 4948 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 18:37:50.833353 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833324 4948 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 18:37:50.833353 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833342 4948 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 18:37:50.833353 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833348 4948 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 18:37:50.833353 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833353 4948 policy_none.go:50] "Start" Nov 03 18:37:50.833465 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833362 4948 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:37:50.833465 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833372 4948 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:37:50.833515 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833493 4948 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 18:37:50.833515 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833500 4948 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 18:37:50.833515 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.833505 4948 policy_none.go:44] "Start" Nov 03 18:37:50.834725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.834689 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:50.834846 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.834741 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:50.834846 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.834757 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:50.837801 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.837775 4948 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:37:50.840345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840331 4948 manager.go:341] "Starting Device Plugin manager" Nov 03 18:37:50.840497 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.840356 4948 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:37:50.840497 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840365 4948 server.go:87] "Starting device plugin registration server" Nov 03 18:37:50.840671 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840659 4948 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:37:50.840720 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840674 4948 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:37:50.840775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840759 4948 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:37:50.840894 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840857 4948 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:37:50.840894 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840895 4948 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:37:50.841042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840902 4948 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:37:50.841042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.840957 4948 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:37:50.840885124Z","UUID":"8f22959e-e157-4b69-8d0a-3644fe0c11d1","Handler":null,"Name":"","Endpoint":""} Nov 03 18:37:50.842222 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.842189 4948 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:37:50.842313 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.842233 4948 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" Nov 03 18:37:50.842783 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.842766 4948 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:37:50.842845 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.842790 4948 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:37:50.845403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.845346 4948 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal"] Nov 03 18:37:50.845507 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.845494 4948 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:50.846398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.846381 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:50.846467 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.846405 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:50.846467 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.846417 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:50.846826 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.846803 4948 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:50.846952 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.846939 4948 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:50.847624 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.847609 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:50.847624 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.847619 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:50.847762 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.847635 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:50.847762 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.847644 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:50.847762 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.847650 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:50.847762 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.847664 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:50.847938 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.847809 4948 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.848011 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.847976 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"3e31283f871e3314b8eda74a4d6e8f72","Type":"ContainerStarted","Data":"9aaf1c360be2cfb4635430176c865f752cb5fe6b735f8f5bf852e0d7d194e0d0"} Nov 03 18:37:50.848062 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848028 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"3e31283f871e3314b8eda74a4d6e8f72","Type":"ContainerStarted","Data":"bf46a0e3fe80fda24f277862ab1f7d3f189ae27e824e04261a253e58c962575e"} Nov 03 18:37:50.848109 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848089 4948 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:50.848156 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848132 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"8a249de0ac9079d1551a521e9f3cd5d3","Type":"ContainerStarted","Data":"c6a981367ab924b0f7c87973ad06271afd99971d1f5d20ddbcd28010d4043b2b"} Nov 03 18:37:50.848215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848158 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"8a249de0ac9079d1551a521e9f3cd5d3","Type":"ContainerDied","Data":"6e559dd37ee6a38b4c4a73dbadc1b90e667a6b08e0bacc4f17163e4dc4882d59"} Nov 03 18:37:50.848215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848176 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" event={"ID":"8a249de0ac9079d1551a521e9f3cd5d3","Type":"ContainerStarted","Data":"2ef4648251a1dbb4dfd56c06237eb6919d118b849fe264cb3acb58b12fd118e8"} Nov 03 18:37:50.848836 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848821 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:50.848908 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848852 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:50.848908 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.848869 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:50.849011 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.848996 4948 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-95.us-west-2.compute.internal\" not found" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.852483 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.852468 4948 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 18:37:50.853004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.852991 4948 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:50.902324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.902303 4948 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:50.903204 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.903188 4948 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:50.903969 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.903954 4948 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:50.914507 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.914480 4948 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.941756 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.941732 4948 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:50.942789 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.942771 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:50.942870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.942803 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:50.942870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.942817 4948 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-95.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:50.942870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.942859 4948 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.953013 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.952965 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31283f871e3314b8eda74a4d6e8f72-config\") pod \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"3e31283f871e3314b8eda74a4d6e8f72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.953013 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.952990 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.953013 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.953008 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.978015 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.977992 4948 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:50.978111 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:50.978040 4948 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:50.978111 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:50.978054 4948 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.053756 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.053717 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.053873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.053780 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.053873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.053802 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31283f871e3314b8eda74a4d6e8f72-config\") pod \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"3e31283f871e3314b8eda74a4d6e8f72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.053873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.053736 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.053873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.053847 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31283f871e3314b8eda74a4d6e8f72-config\") pod \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"3e31283f871e3314b8eda74a4d6e8f72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.053990 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.053871 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a249de0ac9079d1551a521e9f3cd5d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" (UID: \"8a249de0ac9079d1551a521e9f3cd5d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.054544 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.054530 4948 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.054599 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.054591 4948 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.076722 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.076680 4948 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:51.076853 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.076755 4948 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.632617 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.632582 4948 apiserver.go:52] "Watching apiserver" Nov 03 18:37:51.789822 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.789788 4948 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:37:51.790511 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.790476 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-lv47h","openshift-network-console/networking-console-plugin-86c478bb88-drzcf","openshift-multus/network-metrics-daemon-l5g7x","openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h","openshift-ingress-canary/ingress-canary-jphgt","openshift-ovn-kubernetes/ovnkube-node-t92q5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf","openshift-ingress/router-default-7577979dbf-qm2k5","openshift-dns/node-resolver-tlhcd","openshift-insights/insights-runtime-extractor-g96rr","openshift-multus/multus-64c5w","openshift-image-registry/node-ca-hmqxx","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh","openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62","kube-system/kas-connection-checker-685895c7c9-7dks9","kube-system/konnectivity-agent-nl6fg","openshift-network-diagnostics/network-check-target-6bn8h","kube-system/global-pull-secret-syncer-t4dcp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx","openshift-insights/insights-operator-7c4986d449-qqwhz","openshift-console-operator/console-operator-7cd56b8dbc-v5gvn","openshift-dns/dns-default-wt7n9","openshift-network-operator/iptables-alerter-qdczr","kube-system/kas-connection-checker-685895c7c9-pfqbv","kube-system/kas-connection-checker-685895c7c9-d68qq","openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph","openshift-image-registry/image-registry-74945cfd6f-j4qs2","openshift-cluster-node-tuning-operator/tuned-clxql","openshift-image-registry/image-registry-66b9b978fb-5lvgf"] Nov 03 18:37:51.793101 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.793080 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.793443 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.793426 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:51.793773 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.793759 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:51.794740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.794724 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.795999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.795979 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:51.801440 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.801412 4948 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.802654 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.802526 4948 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.806105 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.806087 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:51.806538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.806523 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.806999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.806987 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.807999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.807981 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.816388 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.816372 4948 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:37:51.845884 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.845863 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 18:37:51.845884 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.845879 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 18:37:51.845884 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.845868 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 18:37:51.846344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.846329 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 18:37:51.847247 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847228 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 18:37:51.847247 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847241 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.847344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847234 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.847344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847267 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.847344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847277 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.847344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847293 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 18:37:51.847344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847295 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.847344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.847336 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.848501 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.848475 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n2c2b\"" Nov 03 18:37:51.848583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.848483 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.848583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.848526 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p9fdh\"" Nov 03 18:37:51.848935 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.848919 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:37:51.849009 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.848918 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.849009 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.848974 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 18:37:51.849009 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.848920 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 18:37:51.849171 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849014 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.849171 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849030 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.849171 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849025 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.849171 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849127 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 18:37:51.849171 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849145 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.849403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849184 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.849403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849258 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 18:37:51.849403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849294 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.849403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849326 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 18:37:51.849403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849353 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 18:37:51.849403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849379 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.849403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849380 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849440 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849485 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849518 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849522 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849544 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849558 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849573 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j86t5\"" Nov 03 18:37:51.849641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849631 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 18:37:51.849986 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849673 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 18:37:51.849986 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849781 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-78v26\"" Nov 03 18:37:51.849986 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849806 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.849986 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849888 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 18:37:51.849986 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.849948 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.858894 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.858878 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.858948 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.858904 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.858948 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.858924 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-multus-certs\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.858948 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.858944 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.859084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.858963 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.859084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859072 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysconfig\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.859175 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859104 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffx2\" (UniqueName: \"kubernetes.io/projected/1d027ae4-4c9c-4beb-8134-3995b81d32c1-kube-api-access-mffx2\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.859175 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859118 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.859175 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859126 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:51.859175 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859144 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f023639-2351-4534-8884-8ffdc05a4144-nginx-conf\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:51.859175 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859164 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59c6f41-01de-40f1-9725-1ed9bd6784d6-host-slash\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859187 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cni-binary-copy\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859195 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859211 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-var-lib-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859243 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859249 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859328 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859359 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-registration-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.859398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859397 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmb4\" (UniqueName: \"kubernetes.io/projected/289c8acb-0083-4ec9-ab45-d5e94d47c490-kube-api-access-ddmb4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859452 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-dbus\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859496 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859525 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859551 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8g8\" (UniqueName: \"kubernetes.io/projected/3441fcb6-fc61-446d-a986-79d01440c01b-kube-api-access-7c8g8\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859577 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-systemd-units\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859591 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859601 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-bin\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859650 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-env-overrides\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859675 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.859745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859718 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-k8s-cni-cncf-io\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859757 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859784 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6319a2b4-76e5-42dc-8097-a52570ed7f2c-tmp-dir\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859826 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5583debd-4dae-4fc4-8506-3a452b179306-cert\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859862 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f64f0eb-fed7-496b-92a6-84601b0bb544-serving-cert\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859888 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-tuned\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859873 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859906 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859938 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-etc-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859965 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-kubernetes\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.859974 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-tuned\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860004 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7cq\" (UniqueName: \"kubernetes.io/projected/10497765-1e63-4159-870c-ede32ad1b170-kube-api-access-br7cq\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860022 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860044 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f64f0eb-fed7-496b-92a6-84601b0bb544-serving-cert\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860052 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlpc\" (UniqueName: \"kubernetes.io/projected/d4c6be41-6db5-4be6-8cac-76b00171f6af-kube-api-access-2qlpc\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.860159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860161 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-kubelet-config\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860192 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860221 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6319a2b4-76e5-42dc-8097-a52570ed7f2c-tmp-dir\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860401 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-binary-copy\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860427 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-systemd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860478 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860503 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8cj\" (UniqueName: \"kubernetes.io/projected/edef417a-da49-4ca8-a744-0a3c7569546f-kube-api-access-7b8cj\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860528 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:51.860580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860556 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860622 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6319a2b4-76e5-42dc-8097-a52570ed7f2c-hosts-file\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860647 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3441fcb6-fc61-446d-a986-79d01440c01b-host\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860675 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-lib-modules\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860721 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860742 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbh4\" (UniqueName: \"kubernetes.io/projected/5583debd-4dae-4fc4-8506-3a452b179306-kube-api-access-jdbh4\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860773 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860792 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:51.860877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860875 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860915 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59c6f41-01de-40f1-9725-1ed9bd6784d6-iptables-alerter-script\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860944 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzzp\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860972 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.860996 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-kubelet\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861033 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861071 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861081 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-cnibin\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861114 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76ms\" (UniqueName: \"kubernetes.io/projected/245116c0-8b38-4fd5-868e-fe39362a2113-kube-api-access-m76ms\") pod \"kas-connection-checker-685895c7c9-7dks9\" (UID: \"245116c0-8b38-4fd5-868e-fe39362a2113\") " pod="kube-system/kas-connection-checker-685895c7c9-7dks9" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861218 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfj5x\" (UniqueName: \"kubernetes.io/projected/30383da3-bb10-420f-8f69-587bb4fdacd5-kube-api-access-wfj5x\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:51.861258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861243 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-bin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861259 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861298 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861404 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-systemd\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861453 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-multus\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861486 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3441fcb6-fc61-446d-a986-79d01440c01b-serviceca\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861512 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-ovn\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861538 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861565 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64zv\" (UniqueName: \"kubernetes.io/projected/4a6b250a-51c5-4415-bd6b-4f1275517fdc-kube-api-access-x64zv\") pod \"kas-connection-checker-685895c7c9-d68qq\" (UID: \"4a6b250a-51c5-4415-bd6b-4f1275517fdc\") " pod="kube-system/kas-connection-checker-685895c7c9-d68qq" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861598 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-run\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861636 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-host\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.861683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861656 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-hostroot\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861714 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861756 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861791 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-tmp\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861809 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3441fcb6-fc61-446d-a986-79d01440c01b-serviceca\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861877 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-tmp\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861915 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0ee858f-ae52-4de9-b576-ed63754072f9-agent-certs\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.861963 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.862104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862034 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-daemon-config\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862119 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-stats-auth\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862150 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862184 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkn57\" (UniqueName: \"kubernetes.io/projected/e50d63ba-d363-48f5-a720-880cd7c895d0-kube-api-access-kkn57\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862207 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862222 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862245 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862285 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-netns\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862339 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-stats-auth\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862336 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289c8acb-0083-4ec9-ab45-d5e94d47c490-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862384 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862423 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hdv\" (UniqueName: \"kubernetes.io/projected/175d5971-984f-4e61-bdaa-f1b34609f0e4-kube-api-access-h8hdv\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862449 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-conf-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862460 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289c8acb-0083-4ec9-ab45-d5e94d47c490-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:51.862478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862476 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862513 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862539 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-config\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862623 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg7h\" (UniqueName: \"kubernetes.io/projected/8f64f0eb-fed7-496b-92a6-84601b0bb544-kube-api-access-8xg7h\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862646 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqbv\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862678 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-config\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862681 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6fd\" (UniqueName: \"kubernetes.io/projected/6319a2b4-76e5-42dc-8097-a52570ed7f2c-kube-api-access-2k6fd\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862684 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862737 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862766 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrnd\" (UniqueName: \"kubernetes.io/projected/ce2ce282-8a19-40f3-a97f-a270daedbe66-kube-api-access-pmrnd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862803 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862827 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-modprobe-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862843 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-conf\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862878 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-socket-dir-parent\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862929 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e50d63ba-d363-48f5-a720-880cd7c895d0-serving-cert\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:51.863061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.862976 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-snapshots\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863066 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10497765-1e63-4159-870c-ede32ad1b170-serving-cert\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863098 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e50d63ba-d363-48f5-a720-880cd7c895d0-serving-cert\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863119 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10497765-1e63-4159-870c-ede32ad1b170-snapshots\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863162 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-config\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863236 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10497765-1e63-4159-870c-ede32ad1b170-serving-cert\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863250 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovn-node-metrics-cert\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863340 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee858f-ae52-4de9-b576-ed63754072f9-konnectivity-ca\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863401 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cnibin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863429 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863446 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-netns\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863494 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289c8acb-0083-4ec9-ab45-d5e94d47c490-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863520 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-slash\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863552 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-netd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863599 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-etc-kubernetes\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863626 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863651 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-kubelet\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863673 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289c8acb-0083-4ec9-ab45-d5e94d47c490-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:51.863728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863716 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv95g\" (UniqueName: \"kubernetes.io/projected/066c2936-7cba-4b26-8f35-ba18b055aef2-kube-api-access-jv95g\") pod \"migrator-66df5d496b-ztr62\" (UID: \"066c2936-7cba-4b26-8f35-ba18b055aef2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863749 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7lf\" (UniqueName: \"kubernetes.io/projected/af92784a-e931-457d-a327-5fbba2f37af5-kube-api-access-sp7lf\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863776 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvpg\" (UniqueName: \"kubernetes.io/projected/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-api-access-bcvpg\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863803 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e50d63ba-d363-48f5-a720-880cd7c895d0-config\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863831 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863857 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-script-lib\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863883 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-system-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863908 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-device-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863951 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-sys-fs\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.863996 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c6d42-58af-440c-8c07-915523a8a3bb-crio-socket\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864025 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e6c6d42-58af-440c-8c07-915523a8a3bb-data-volume\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864032 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e50d63ba-d363-48f5-a720-880cd7c895d0-config\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864098 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e6c6d42-58af-440c-8c07-915523a8a3bb-data-volume\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864133 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edef417a-da49-4ca8-a744-0a3c7569546f-config-volume\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864172 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10497765-1e63-4159-870c-ede32ad1b170-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864179 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-system-cni-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864221 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjs9k\" (UniqueName: \"kubernetes.io/projected/a59c6f41-01de-40f1-9725-1ed9bd6784d6-kube-api-access-gjs9k\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864247 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-default-certificate\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.864364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864297 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbnd\" (UniqueName: \"kubernetes.io/projected/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-kube-api-access-rxbnd\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864404 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjg9j\" (UniqueName: \"kubernetes.io/projected/c3e13ce6-21d8-4d9c-b589-46699c28b856-kube-api-access-gjg9j\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864439 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-sys\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864458 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-var-lib-kubelet\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864482 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-default-certificate\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864517 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5nn\" (UniqueName: \"kubernetes.io/projected/d0f084d2-6540-44ee-ac2b-c8e6df92a050-kube-api-access-mb5nn\") pod \"kas-connection-checker-685895c7c9-pfqbv\" (UID: \"d0f084d2-6540-44ee-ac2b-c8e6df92a050\") " pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864564 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b52\" (UniqueName: \"kubernetes.io/projected/07926d6b-bf32-412b-89e5-26c0778a793f-kube-api-access-75b52\") pod \"network-check-source-6f78d59f99-tqhrb\" (UID: \"07926d6b-bf32-412b-89e5-26c0778a793f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864614 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-node-log\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864637 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864666 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64s6\" (UniqueName: \"kubernetes.io/projected/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-kube-api-access-c64s6\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864714 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-log-socket\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864743 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqqt\" (UniqueName: \"kubernetes.io/projected/fe291b57-902f-430e-91ed-d373c931c885-kube-api-access-2jqqt\") pod \"volume-data-source-validator-58c9f5f46d-q8cd2\" (UID: \"fe291b57-902f-430e-91ed-d373c931c885\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864768 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edef417a-da49-4ca8-a744-0a3c7569546f-tmp-dir\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864818 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-tmp\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864842 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edef417a-da49-4ca8-a744-0a3c7569546f-tmp-dir\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864852 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-socket-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864884 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.864902 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175d5971-984f-4e61-bdaa-f1b34609f0e4-tmp\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.899873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.899826 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 18:37:51.901774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.901753 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 18:37:51.901924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.901910 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 18:37:51.904510 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.904277 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f64f0eb-fed7-496b-92a6-84601b0bb544-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.904974 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.904824 4948 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:51.904974 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.904877 4948 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.906344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.904824 4948 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:51.906344 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.905366 4948 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal" Nov 03 18:37:51.906483 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.906357 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.910032 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.910013 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:51.910201 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.910184 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.922901 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.922880 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmb4\" (UniqueName: \"kubernetes.io/projected/289c8acb-0083-4ec9-ab45-d5e94d47c490-kube-api-access-ddmb4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-546xx\" (UID: \"289c8acb-0083-4ec9-ab45-d5e94d47c490\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" Nov 03 18:37:51.925164 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.925141 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8g8\" (UniqueName: \"kubernetes.io/projected/3441fcb6-fc61-446d-a986-79d01440c01b-kube-api-access-7c8g8\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:51.925281 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.925248 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7cq\" (UniqueName: \"kubernetes.io/projected/10497765-1e63-4159-870c-ede32ad1b170-kube-api-access-br7cq\") pod \"insights-operator-7c4986d449-qqwhz\" (UID: \"10497765-1e63-4159-870c-ede32ad1b170\") " pod="openshift-insights/insights-operator-7c4986d449-qqwhz" Nov 03 18:37:51.925642 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.925625 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.932194 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.932178 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5nn\" (UniqueName: \"kubernetes.io/projected/d0f084d2-6540-44ee-ac2b-c8e6df92a050-kube-api-access-mb5nn\") pod \"kas-connection-checker-685895c7c9-pfqbv\" (UID: \"d0f084d2-6540-44ee-ac2b-c8e6df92a050\") " pod="kube-system/kas-connection-checker-685895c7c9-pfqbv" Nov 03 18:37:51.932918 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.932903 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqbv\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.933009 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.932993 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbnd\" (UniqueName: \"kubernetes.io/projected/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-kube-api-access-rxbnd\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.933353 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.933339 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.933908 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.933892 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqqt\" (UniqueName: \"kubernetes.io/projected/fe291b57-902f-430e-91ed-d373c931c885-kube-api-access-2jqqt\") pod \"volume-data-source-validator-58c9f5f46d-q8cd2\" (UID: \"fe291b57-902f-430e-91ed-d373c931c885\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q8cd2" Nov 03 18:37:51.936661 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.936647 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkn57\" (UniqueName: \"kubernetes.io/projected/e50d63ba-d363-48f5-a720-880cd7c895d0-kube-api-access-kkn57\") pod \"service-ca-operator-544d749c8c-cbtph\" (UID: \"e50d63ba-d363-48f5-a720-880cd7c895d0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" Nov 03 18:37:51.936722 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.936693 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76ms\" (UniqueName: \"kubernetes.io/projected/245116c0-8b38-4fd5-868e-fe39362a2113-kube-api-access-m76ms\") pod \"kas-connection-checker-685895c7c9-7dks9\" (UID: \"245116c0-8b38-4fd5-868e-fe39362a2113\") " pod="kube-system/kas-connection-checker-685895c7c9-7dks9" Nov 03 18:37:51.937500 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.937484 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg7h\" (UniqueName: \"kubernetes.io/projected/8f64f0eb-fed7-496b-92a6-84601b0bb544-kube-api-access-8xg7h\") pod \"console-operator-7cd56b8dbc-v5gvn\" (UID: \"8f64f0eb-fed7-496b-92a6-84601b0bb544\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:51.937549 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.937536 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64zv\" (UniqueName: \"kubernetes.io/projected/4a6b250a-51c5-4415-bd6b-4f1275517fdc-kube-api-access-x64zv\") pod \"kas-connection-checker-685895c7c9-d68qq\" (UID: \"4a6b250a-51c5-4415-bd6b-4f1275517fdc\") " pod="kube-system/kas-connection-checker-685895c7c9-d68qq" Nov 03 18:37:51.937978 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.937962 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfj5x\" (UniqueName: \"kubernetes.io/projected/30383da3-bb10-420f-8f69-587bb4fdacd5-kube-api-access-wfj5x\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:51.938010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.937987 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvpg\" (UniqueName: \"kubernetes.io/projected/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-api-access-bcvpg\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.938689 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.938674 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64s6\" (UniqueName: \"kubernetes.io/projected/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-kube-api-access-c64s6\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:51.942325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.942308 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzzp\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.947583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947557 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.947714 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947687 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:37:51.947850 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947836 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:37:51.947949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947936 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:37:51.948001 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947955 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.948001 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947961 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:37:51.948001 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947979 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:37:51.948112 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948000 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:37:51.948112 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.947989 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.948340 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948325 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.948427 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948415 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:37:51.948490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948464 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.948583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948569 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 18:37:51.948630 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948597 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 18:37:51.948630 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948615 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.948740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948647 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d4mhq\"" Nov 03 18:37:51.948740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948681 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.948740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948714 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.948740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948721 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:37:51.948740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948718 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.948949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948753 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.948949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948820 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:37:51.948949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948822 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lzn26\"" Nov 03 18:37:51.948949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948879 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:37:51.948949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948917 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:37:51.948949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.948933 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkfh8\"" Nov 03 18:37:51.949160 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949103 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.949160 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949114 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dvl9\"" Nov 03 18:37:51.949160 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949136 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:37:51.949160 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949149 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949172 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949193 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88hbk\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949173 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949211 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczj\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949214 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cxqpk\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949215 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4grw\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949234 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949226 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949253 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fft52\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949260 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949261 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z9j9b\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949281 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949283 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zjfk4\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949298 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949303 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949173 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949333 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949342 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949356 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-btttq\"" Nov 03 18:37:51.949351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949362 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949367 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxjph\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949371 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc7sj\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949403 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f023639-2351-4534-8884-8ffdc05a4144-nginx-conf\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949478 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cni-binary-copy\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949632 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949632 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bgcqc\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949671 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-77qxk\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949687 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r88kn\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949739 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949807 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zsb8x\"" Nov 03 18:37:51.949954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.949895 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-env-overrides\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.950974 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.950961 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-cni-binary-copy\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.951033 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.950968 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5583debd-4dae-4fc4-8506-3a452b179306-cert\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:51.951111 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.951096 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e6c6d42-58af-440c-8c07-915523a8a3bb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.951169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.951154 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59c6f41-01de-40f1-9725-1ed9bd6784d6-iptables-alerter-script\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:51.951594 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.951576 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbh4\" (UniqueName: \"kubernetes.io/projected/5583debd-4dae-4fc4-8506-3a452b179306-kube-api-access-jdbh4\") pod \"ingress-canary-jphgt\" (UID: \"5583debd-4dae-4fc4-8506-3a452b179306\") " pod="openshift-ingress-canary/ingress-canary-jphgt" Nov 03 18:37:51.952156 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.952135 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c6be41-6db5-4be6-8cac-76b00171f6af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.952280 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.952267 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-daemon-config\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.952338 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.952299 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0ee858f-ae52-4de9-b576-ed63754072f9-agent-certs\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:51.952577 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.952559 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b52\" (UniqueName: \"kubernetes.io/projected/07926d6b-bf32-412b-89e5-26c0778a793f-kube-api-access-75b52\") pod \"network-check-source-6f78d59f99-tqhrb\" (UID: \"07926d6b-bf32-412b-89e5-26c0778a793f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tqhrb" Nov 03 18:37:51.952978 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.952961 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7lf\" (UniqueName: \"kubernetes.io/projected/af92784a-e931-457d-a327-5fbba2f37af5-kube-api-access-sp7lf\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.952978 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.952969 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjg9j\" (UniqueName: \"kubernetes.io/projected/c3e13ce6-21d8-4d9c-b589-46699c28b856-kube-api-access-gjg9j\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:51.953073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.952996 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffx2\" (UniqueName: \"kubernetes.io/projected/1d027ae4-4c9c-4beb-8134-3995b81d32c1-kube-api-access-mffx2\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.953412 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.953398 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovn-node-metrics-cert\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.953481 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.953454 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-config\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.953547 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.953534 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee858f-ae52-4de9-b576-ed63754072f9-konnectivity-ca\") pod \"konnectivity-agent-nl6fg\" (UID: \"d0ee858f-ae52-4de9-b576-ed63754072f9\") " pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:37:51.953671 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.953653 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hdv\" (UniqueName: \"kubernetes.io/projected/175d5971-984f-4e61-bdaa-f1b34609f0e4-kube-api-access-h8hdv\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.953901 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.953884 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjs9k\" (UniqueName: \"kubernetes.io/projected/a59c6f41-01de-40f1-9725-1ed9bd6784d6-kube-api-access-gjs9k\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:51.954131 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.954115 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce2ce282-8a19-40f3-a97f-a270daedbe66-ovnkube-script-lib\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.954219 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.954203 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8cj\" (UniqueName: \"kubernetes.io/projected/edef417a-da49-4ca8-a744-0a3c7569546f-kube-api-access-7b8cj\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.954346 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.954329 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlpc\" (UniqueName: \"kubernetes.io/projected/d4c6be41-6db5-4be6-8cac-76b00171f6af-kube-api-access-2qlpc\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.954346 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.954340 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edef417a-da49-4ca8-a744-0a3c7569546f-config-volume\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.955739 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.955723 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-original-pull-secret\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.958049 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.958030 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6fd\" (UniqueName: \"kubernetes.io/projected/6319a2b4-76e5-42dc-8097-a52570ed7f2c-kube-api-access-2k6fd\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:51.958223 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.958202 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrnd\" (UniqueName: \"kubernetes.io/projected/ce2ce282-8a19-40f3-a97f-a270daedbe66-kube-api-access-pmrnd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.958361 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.958348 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fj8\" (UniqueName: \"kubernetes.io/projected/6b7d3c74-e08e-44b1-ab35-ced1363278f0-kube-api-access-m5fj8\") pod \"network-check-target-6bn8h\" (UID: \"6b7d3c74-e08e-44b1-ab35-ced1363278f0\") " pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:51.958731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.958717 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv95g\" (UniqueName: \"kubernetes.io/projected/066c2936-7cba-4b26-8f35-ba18b055aef2-kube-api-access-jv95g\") pod \"migrator-66df5d496b-ztr62\" (UID: \"066c2936-7cba-4b26-8f35-ba18b055aef2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" Nov 03 18:37:51.965232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965216 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c6d42-58af-440c-8c07-915523a8a3bb-crio-socket\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.965278 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965240 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-system-cni-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.965278 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965257 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-sys\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965278 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965271 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-var-lib-kubelet\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965322 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-system-cni-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965331 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-sys\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965360 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-node-log\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965374 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c6d42-58af-440c-8c07-915523a8a3bb-crio-socket\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965388 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-var-lib-kubelet\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965390 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-log-socket\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965401 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-node-log\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965416 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-socket-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965427 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-log-socket\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965438 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-multus-certs\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965454 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysconfig\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965476 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59c6f41-01de-40f1-9725-1ed9bd6784d6-host-slash\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965492 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-multus-certs\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965506 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-var-lib-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965530 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965537 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59c6f41-01de-40f1-9725-1ed9bd6784d6-host-slash\") pod \"iptables-alerter-qdczr\" (UID: \"a59c6f41-01de-40f1-9725-1ed9bd6784d6\") " pod="openshift-network-operator/iptables-alerter-qdczr" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965550 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-var-lib-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965532 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-socket-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965535 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysconfig\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965552 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965594 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-registration-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965618 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-dbus\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.965623 4948 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965645 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-systemd-units\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965671 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-bin\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965683 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-systemd-units\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965710 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-k8s-cni-cncf-io\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965715 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-registration-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965723 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-bin\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965712 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-dbus\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965645 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965746 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-k8s-cni-cncf-io\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.965742 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.465727568 +0000 UTC m=+1.948723478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:51.965768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965777 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-etc-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965807 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-etc-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965806 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-kubernetes\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965843 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-kubelet-config\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965850 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-kubernetes\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965869 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965889 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-systemd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965905 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965907 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8da56f0f-6ab9-4eb3-ad1d-35ce98845ada-kubelet-config\") pod \"global-pull-secret-syncer-t4dcp\" (UID: \"8da56f0f-6ab9-4eb3-ad1d-35ce98845ada\") " pod="kube-system/global-pull-secret-syncer-t4dcp" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965935 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965951 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6319a2b4-76e5-42dc-8097-a52570ed7f2c-hosts-file\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965967 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3441fcb6-fc61-446d-a986-79d01440c01b-host\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965987 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-lib-modules\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.965990 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-systemd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.965967 4948 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.965971 4948 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966007 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3441fcb6-fc61-446d-a986-79d01440c01b-host\") pod \"node-ca-hmqxx\" (UID: \"3441fcb6-fc61-446d-a986-79d01440c01b\") " pod="openshift-image-registry/node-ca-hmqxx" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966018 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6319a2b4-76e5-42dc-8097-a52570ed7f2c-hosts-file\") pod \"node-resolver-tlhcd\" (UID: \"6319a2b4-76e5-42dc-8097-a52570ed7f2c\") " pod="openshift-dns/node-resolver-tlhcd" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966020 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966058 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466045782 +0000 UTC m=+1.949041687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966079 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966085 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466072872 +0000 UTC m=+1.949068765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966088 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966092 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-lib-modules\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966102 4948 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966127 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966172 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466161719 +0000 UTC m=+1.949157640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966179 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966184 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466177928 +0000 UTC m=+1.949173845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : secret "metrics-daemon-secret" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966204 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-kubelet\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966221 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966237 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-cnibin\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966253 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-bin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966271 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966276 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-kubelet\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966289 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966297 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-cnibin\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966303 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-systemd\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966311 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-bin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966313 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466301518 +0000 UTC m=+1.949297427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966333 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-systemd\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966336 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-openvswitch\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966347 4948 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966358 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-multus\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966377 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-ovn\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966393 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-run\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966407 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-host\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966429 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-hostroot\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966440 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466431156 +0000 UTC m=+1.949427053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : secret "insights-runtime-extractor-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966441 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-run-ovn\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966456 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-var-lib-cni-multus\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966445 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-run\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966465 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966473 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-host\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966485 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-hostroot\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966486 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966494 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966516 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966519 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966531 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-netns\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966546 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966560 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-conf-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966575 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966587 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966594 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966619 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-host-run-netns\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966630 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966637 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966639 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-conf-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966650 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-modprobe-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966619 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c6be41-6db5-4be6-8cac-76b00171f6af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lv47h\" (UID: \"d4c6be41-6db5-4be6-8cac-76b00171f6af\") " pod="openshift-multus/multus-additional-cni-plugins-lv47h" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966645 4948 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966691 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966713 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466686751 +0000 UTC m=+1.949682678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966717 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966732 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-conf\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966734 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-modprobe-d\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966750 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.466739863 +0000 UTC m=+1.949735754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966779 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-socket-dir-parent\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966797 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cnibin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966812 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-netns\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966828 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-slash\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966828 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/175d5971-984f-4e61-bdaa-f1b34609f0e4-etc-sysctl-conf\") pod \"tuned-clxql\" (UID: \"175d5971-984f-4e61-bdaa-f1b34609f0e4\") " pod="openshift-cluster-node-tuning-operator/tuned-clxql" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966842 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-netd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966853 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-cnibin\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966861 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-multus-socket-dir-parent\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966867 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-run-netns\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.979040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966869 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-slash\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966881 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-cni-netd\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966871 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-etc-kubernetes\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966900 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-etc-kubernetes\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966921 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966942 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-kubelet\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966959 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-system-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966966 4948 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.966974 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-device-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:51.966994 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.46698512 +0000 UTC m=+1.949981013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.967006 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-sys-fs\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.967007 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce2ce282-8a19-40f3-a97f-a270daedbe66-host-kubelet\") pod \"ovnkube-node-t92q5\" (UID: \"ce2ce282-8a19-40f3-a97f-a270daedbe66\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.967010 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-device-dir\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.967023 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d027ae4-4c9c-4beb-8134-3995b81d32c1-system-cni-dir\") pod \"multus-64c5w\" (UID: \"1d027ae4-4c9c-4beb-8134-3995b81d32c1\") " pod="openshift-multus/multus-64c5w" Nov 03 18:37:51.981374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:51.967069 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af92784a-e931-457d-a327-5fbba2f37af5-sys-fs\") pod \"aws-ebs-csi-driver-node-4zg2h\" (UID: \"af92784a-e931-457d-a327-5fbba2f37af5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zg2h" Nov 03 18:37:52.094784 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.094759 4948 scope.go:122] "RemoveContainer" containerID="9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb" Nov 03 18:37:52.110062 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.110042 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:37:52.471256 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471225 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:52.471256 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471260 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471286 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471322 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471349 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471368 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471375 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471389 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471394 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471447 4948 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471464 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471446888 +0000 UTC m=+2.954442800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471468 4948 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471475 4948 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471483 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471487 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:52.471490 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471498 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471379 4948 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471542 4948 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471453 4948 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471500 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471487278 +0000 UTC m=+2.954483172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471618 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471597763 +0000 UTC m=+2.954593666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471644 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471633915 +0000 UTC m=+2.954629810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : secret "metrics-daemon-secret" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471660 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471652855 +0000 UTC m=+2.954648747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471672 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471667414 +0000 UTC m=+2.954663305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471685 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471680658 +0000 UTC m=+2.954676549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471719 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471713269 +0000 UTC m=+2.954709161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471734 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.471754 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471831 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.471820807 +0000 UTC m=+2.954816711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471832 4948 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:37:52.471924 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.471861 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.47185502 +0000 UTC m=+2.954850952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : secret "insights-runtime-extractor-tls" not found Nov 03 18:37:52.810050 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.809971 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/2.log" Nov 03 18:37:52.810474 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.810465 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/1.log" Nov 03 18:37:52.810521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.810492 4948 generic.go:356] "Generic (PLEG): container finished" podID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerID="ac22fcb4eba80f6163068ec4dce5a192e88956bcedf6385396b9b30446c7bd2d" exitCode=255 Nov 03 18:37:52.810596 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.810575 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" event={"ID":"8f64f0eb-fed7-496b-92a6-84601b0bb544","Type":"ContainerDied","Data":"ac22fcb4eba80f6163068ec4dce5a192e88956bcedf6385396b9b30446c7bd2d"} Nov 03 18:37:52.810681 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.810667 4948 scope.go:122] "RemoveContainer" containerID="9e46d2fba0f765c343cbc20b15abb62a97437893deb228fb1d8c76491eddfadb" Nov 03 18:37:52.811152 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.811131 4948 scope.go:122] "RemoveContainer" containerID="ac22fcb4eba80f6163068ec4dce5a192e88956bcedf6385396b9b30446c7bd2d" Nov 03 18:37:52.811317 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:52.811299 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:37:52.984739 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:52.984711 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:53.481194 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481166 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:53.481345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481201 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:53.481345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481238 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:53.481345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481293 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:53.481345 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481316 4948 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:53.481345 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481323 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:53.481345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481335 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:53.481345 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481347 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481366 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481396 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481415 4948 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481426 4948 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481433 4948 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481415 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481395876 +0000 UTC m=+4.964391786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481461 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481472 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481475 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481460472 +0000 UTC m=+4.964456377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481487 4948 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481494 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481485335 +0000 UTC m=+4.964481227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481571 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481555466 +0000 UTC m=+4.964551358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481593 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.48158528 +0000 UTC m=+4.964581172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481608 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.48160342 +0000 UTC m=+4.964599312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481629 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481624176 +0000 UTC m=+4.964620067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:53.481658 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481644 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:53.482208 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481671 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:53.482208 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.481692 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:53.482208 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481762 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481754836 +0000 UTC m=+4.964750766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:53.482208 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481796 4948 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 18:37:53.482208 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481814 4948 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:37:53.482208 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481845 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481838732 +0000 UTC m=+4.964834623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : secret "insights-runtime-extractor-tls" not found Nov 03 18:37:53.482208 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:53.481856 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.481851121 +0000 UTC m=+4.964847013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : secret "metrics-daemon-secret" not found Nov 03 18:37:53.817283 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.817202 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ztr62" event={"ID":"066c2936-7cba-4b26-8f35-ba18b055aef2","Type":"ContainerStarted","Data":"4093b8308fd09244aac72e54dc5230c7b2792c2164a8e1ceabbdd0d904518535"} Nov 03 18:37:53.818650 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:53.818629 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/2.log" Nov 03 18:37:54.006582 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:54.006553 4948 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:37:54.006899 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:54.006885 4948 scope.go:122] "RemoveContainer" containerID="ac22fcb4eba80f6163068ec4dce5a192e88956bcedf6385396b9b30446c7bd2d" Nov 03 18:37:54.007074 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:54.007056 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:37:54.954384 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:54.954356 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:37:54.954792 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:54.954555 4948 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:37:55.499127 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499080 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499141 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499165 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499182 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499202 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499221 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499224 4948 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499235 4948 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499241 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499287 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499270027 +0000 UTC m=+8.982265929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499300 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499310 4948 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499316 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499309029 +0000 UTC m=+8.982304924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:55.499325 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499317 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499334 4948 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499353 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499337163 +0000 UTC m=+8.982333092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499363 4948 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499381 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls podName:4e6c6d42-58af-440c-8c07-915523a8a3bb nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499374339 +0000 UTC m=+8.982370231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g96rr" (UID: "4e6c6d42-58af-440c-8c07-915523a8a3bb") : secret "insights-runtime-extractor-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499395 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499388842 +0000 UTC m=+8.982384735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499408 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert podName:0f023639-2351-4534-8884-8ffdc05a4144 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499400808 +0000 UTC m=+8.982396702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-drzcf" (UID: "0f023639-2351-4534-8884-8ffdc05a4144") : secret "networking-console-plugin-cert" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499441 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499420623 +0000 UTC m=+8.982416528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : secret "metrics-daemon-secret" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499490 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499527 4948 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499550 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499556 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls podName:30383da3-bb10-420f-8f69-587bb4fdacd5 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499547828 +0000 UTC m=+8.982543725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f86dh" (UID: "30383da3-bb10-420f-8f69-587bb4fdacd5") : secret "samples-operator-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499593 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:55.499595 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499600 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499646 4948 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499655 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499643235 +0000 UTC m=+8.982639142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:55.499781 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:55.499673 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.499664441 +0000 UTC m=+8.982660332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:57.101408 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:57.101379 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlhcd_6319a2b4-76e5-42dc-8097-a52570ed7f2c/dns-node-resolver/0.log" Nov 03 18:37:57.900085 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:57.900057 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hmqxx_3441fcb6-fc61-446d-a986-79d01440c01b/node-ca/0.log" Nov 03 18:37:58.902510 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:58.902476 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jphgt_5583debd-4dae-4fc4-8506-3a452b179306/serve-healthcheck-canary/0.log" Nov 03 18:37:59.300846 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.300803 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ztr62_066c2936-7cba-4b26-8f35-ba18b055aef2/migrator/0.log" Nov 03 18:37:59.501582 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.501550 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-546xx_289c8acb-0083-4ec9-ab45-d5e94d47c490/kube-storage-version-migrator-operator/0.log" Nov 03 18:37:59.531775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.531748 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:37:59.533345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533324 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:59.533450 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533373 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:37:59.533450 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533411 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:59.533450 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533435 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533457 4948 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533464 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533495 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533510 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.533492644 +0000 UTC m=+17.016488550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : secret "router-metrics-certs-default" not found Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533543 4948 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533556 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533565 4948 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533570 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74945cfd6f-j4qs2: secret "image-registry-tls" not found Nov 03 18:37:59.533604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533549 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533601 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls podName:fd5c02ff-a13f-4d45-bb84-df74ec8fd86f nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.533588284 +0000 UTC m=+17.016584181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-75cdf" (UID: "fd5c02ff-a13f-4d45-bb84-df74ec8fd86f") : secret "cluster-monitoring-operator-tls" not found Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533657 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs podName:c3e13ce6-21d8-4d9c-b589-46699c28b856 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.533634624 +0000 UTC m=+17.016630518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs") pod "network-metrics-daemon-l5g7x" (UID: "c3e13ce6-21d8-4d9c-b589-46699c28b856") : secret "metrics-daemon-secret" not found Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533678 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls podName:8aae4dec-41f5-4cc5-809d-cc496fdb1d77 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.533668265 +0000 UTC m=+17.016664163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls") pod "image-registry-74945cfd6f-j4qs2" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77") : secret "image-registry-tls" not found Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533717 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle podName:0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.533693763 +0000 UTC m=+17.016689658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle") pod "router-default-7577979dbf-qm2k5" (UID: "0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9") : configmap references non-existent config key: service-ca.crt Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533782 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533822 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533855 4948 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.533870 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533970 4948 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 18:37:59.533997 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.533985 4948 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b9b978fb-5lvgf: secret "image-registry-tls" not found Nov 03 18:37:59.534339 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.534019 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls podName:d16fa79d-4ad1-4fb4-b60b-915f0bbcb677 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.534009488 +0000 UTC m=+17.017005383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls") pod "image-registry-66b9b978fb-5lvgf" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677") : secret "image-registry-tls" not found Nov 03 18:37:59.534339 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.534036 4948 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 18:37:59.534339 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:37:59.534067 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls podName:edef417a-da49-4ca8-a744-0a3c7569546f nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.534059482 +0000 UTC m=+17.017055373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls") pod "dns-default-wt7n9" (UID: "edef417a-da49-4ca8-a744-0a3c7569546f") : secret "dns-default-metrics-tls" not found Nov 03 18:37:59.536032 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.536008 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e6c6d42-58af-440c-8c07-915523a8a3bb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g96rr\" (UID: \"4e6c6d42-58af-440c-8c07-915523a8a3bb\") " pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:37:59.536134 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.536079 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f023639-2351-4534-8884-8ffdc05a4144-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-drzcf\" (UID: \"0f023639-2351-4534-8884-8ffdc05a4144\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:59.536134 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.536120 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30383da3-bb10-420f-8f69-587bb4fdacd5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f86dh\" (UID: \"30383da3-bb10-420f-8f69-587bb4fdacd5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:59.594312 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.594238 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" Nov 03 18:37:59.597285 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.597267 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" Nov 03 18:37:59.609231 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:37:59.609207 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g96rr" Nov 03 18:38:00.003376 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.003340 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:38:00.003796 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.003767 4948 scope.go:122] "RemoveContainer" containerID="ac22fcb4eba80f6163068ec4dce5a192e88956bcedf6385396b9b30446c7bd2d" Nov 03 18:38:00.003963 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:38:00.003943 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:38:00.070738 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.070689 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-drzcf"] Nov 03 18:38:00.073857 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:00.073828 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f023639_2351_4534_8884_8ffdc05a4144.slice/crio-4872eab5b491b9d8a086c8679ba7cc37331c652e64e77e998ad40c4a1e39f971 WatchSource:0}: Error finding container 4872eab5b491b9d8a086c8679ba7cc37331c652e64e77e998ad40c4a1e39f971: Status 404 returned error can't find the container with id 4872eab5b491b9d8a086c8679ba7cc37331c652e64e77e998ad40c4a1e39f971 Nov 03 18:38:00.074196 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.074170 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh"] Nov 03 18:38:00.074552 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.074526 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g96rr"] Nov 03 18:38:00.121620 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:00.121580 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6c6d42_58af_440c_8c07_915523a8a3bb.slice/crio-96b7a1ffb0f818b1c4b37d190e35fdd35ca550a2f9276aa3552816536a7532da WatchSource:0}: Error finding container 96b7a1ffb0f818b1c4b37d190e35fdd35ca550a2f9276aa3552816536a7532da: Status 404 returned error can't find the container with id 96b7a1ffb0f818b1c4b37d190e35fdd35ca550a2f9276aa3552816536a7532da Nov 03 18:38:00.840988 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.840950 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g96rr" event={"ID":"4e6c6d42-58af-440c-8c07-915523a8a3bb","Type":"ContainerStarted","Data":"d5f4c5e57f8de81865d2ca8ddf9c0935b14a8014e7f83b30d804d88e677bba1b"} Nov 03 18:38:00.841163 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.841024 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g96rr" event={"ID":"4e6c6d42-58af-440c-8c07-915523a8a3bb","Type":"ContainerStarted","Data":"96b7a1ffb0f818b1c4b37d190e35fdd35ca550a2f9276aa3552816536a7532da"} Nov 03 18:38:00.842080 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.842058 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" event={"ID":"0f023639-2351-4534-8884-8ffdc05a4144","Type":"ContainerStarted","Data":"4872eab5b491b9d8a086c8679ba7cc37331c652e64e77e998ad40c4a1e39f971"} Nov 03 18:38:00.843126 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:00.843098 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" event={"ID":"30383da3-bb10-420f-8f69-587bb4fdacd5","Type":"ContainerStarted","Data":"e64231bf9a0d0e26cf7e354684391ae51b17ea49af97fa4da1f752e16d6d2e6d"} Nov 03 18:38:01.849218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:01.849132 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g96rr" event={"ID":"4e6c6d42-58af-440c-8c07-915523a8a3bb","Type":"ContainerStarted","Data":"5e8081cfde31b1f82076a3fca3d1d1b87de476bd010dbc4bbc58464b987ae113"} Nov 03 18:38:02.746979 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.746544 4948 scope.go:122] "RemoveContainer" containerID="ac22fcb4eba80f6163068ec4dce5a192e88956bcedf6385396b9b30446c7bd2d" Nov 03 18:38:02.855359 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.855282 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/2.log" Nov 03 18:38:02.855774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.855404 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" event={"ID":"8f64f0eb-fed7-496b-92a6-84601b0bb544","Type":"ContainerStarted","Data":"a02ae980477e7bdb3d99cea2143d455de5371f7e6bf1c5d65955d2076739b985"} Nov 03 18:38:02.855774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.855726 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:38:02.857577 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.857532 4948 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-v5gvn container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Nov 03 18:38:02.857680 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.857602 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Nov 03 18:38:02.858633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.858606 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-drzcf" event={"ID":"0f023639-2351-4534-8884-8ffdc05a4144","Type":"ContainerStarted","Data":"1ed73227879f69fb7df6b0de8def979efe9aaaa3e444c4b2f266d92b4e62dcb8"} Nov 03 18:38:02.860948 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.860926 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" event={"ID":"30383da3-bb10-420f-8f69-587bb4fdacd5","Type":"ContainerStarted","Data":"763aab44e1535c1721f8716b094278386f0f7318b8cf95b78f7aed1bb2ca788f"} Nov 03 18:38:02.861051 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:02.860970 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f86dh" event={"ID":"30383da3-bb10-420f-8f69-587bb4fdacd5","Type":"ContainerStarted","Data":"739b9eb43b7007c983606166dcffa7c36ca720babb6054fd0bf34693f17a40ba"} Nov 03 18:38:03.866249 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:03.866214 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g96rr" event={"ID":"4e6c6d42-58af-440c-8c07-915523a8a3bb","Type":"ContainerStarted","Data":"b4c38c8aa71c2559be8435fe06f34c11f90c41cadbeeb1a3d66ecdbc02e40e87"} Nov 03 18:38:03.867830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:03.867805 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:38:03.868205 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:03.868190 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/2.log" Nov 03 18:38:03.868266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:03.868222 4948 generic.go:356] "Generic (PLEG): container finished" podID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerID="a02ae980477e7bdb3d99cea2143d455de5371f7e6bf1c5d65955d2076739b985" exitCode=255 Nov 03 18:38:03.868311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:03.868252 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" event={"ID":"8f64f0eb-fed7-496b-92a6-84601b0bb544","Type":"ContainerDied","Data":"a02ae980477e7bdb3d99cea2143d455de5371f7e6bf1c5d65955d2076739b985"} Nov 03 18:38:03.868311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:03.868307 4948 scope.go:122] "RemoveContainer" containerID="ac22fcb4eba80f6163068ec4dce5a192e88956bcedf6385396b9b30446c7bd2d" Nov 03 18:38:03.868494 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:03.868475 4948 scope.go:122] "RemoveContainer" containerID="a02ae980477e7bdb3d99cea2143d455de5371f7e6bf1c5d65955d2076739b985" Nov 03 18:38:03.868725 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:38:03.868684 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:38:04.007500 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:04.007452 4948 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:38:04.873307 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:04.873283 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:38:04.873660 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:04.873624 4948 scope.go:122] "RemoveContainer" containerID="a02ae980477e7bdb3d99cea2143d455de5371f7e6bf1c5d65955d2076739b985" Nov 03 18:38:04.873851 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:38:04.873832 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:38:05.877521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:05.877482 4948 scope.go:122] "RemoveContainer" containerID="a02ae980477e7bdb3d99cea2143d455de5371f7e6bf1c5d65955d2076739b985" Nov 03 18:38:05.878007 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:38:05.877833 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5gvn_openshift-console-operator(8f64f0eb-fed7-496b-92a6-84601b0bb544)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" Nov 03 18:38:07.612775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.612735 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:38:07.612775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.612778 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:07.613348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.612816 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:38:07.613348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.612841 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:38:07.613348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.612876 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:07.613348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.612904 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:38:07.613348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.613009 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:38:07.613608 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.613578 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-service-ca-bundle\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:07.615893 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.615864 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e13ce6-21d8-4d9c-b589-46699c28b856-metrics-certs\") pod \"network-metrics-daemon-l5g7x\" (UID: \"c3e13ce6-21d8-4d9c-b589-46699c28b856\") " pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:38:07.616034 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.615960 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5c02ff-a13f-4d45-bb84-df74ec8fd86f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-75cdf\" (UID: \"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:38:07.616034 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.616023 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"image-registry-66b9b978fb-5lvgf\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:38:07.616389 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.616370 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edef417a-da49-4ca8-a744-0a3c7569546f-metrics-tls\") pod \"dns-default-wt7n9\" (UID: \"edef417a-da49-4ca8-a744-0a3c7569546f\") " pod="openshift-dns/dns-default-wt7n9" Nov 03 18:38:07.616389 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.616382 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"image-registry-74945cfd6f-j4qs2\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:38:07.616452 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.616379 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9-metrics-certs\") pod \"router-default-7577979dbf-qm2k5\" (UID: \"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9\") " pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:07.694328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.694277 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:07.694480 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.694368 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" Nov 03 18:38:07.695336 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.695319 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:38:07.707304 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.707287 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5g7x" Nov 03 18:38:07.707437 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.707412 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt7n9" Nov 03 18:38:07.707547 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:07.707472 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:38:08.191556 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.191519 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l5g7x"] Nov 03 18:38:08.194568 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:08.194536 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e13ce6_21d8_4d9c_b589_46699c28b856.slice/crio-7521e434794ff9fc02662ad894fe4afecdf6c33800c996d7314cf501f7764319 WatchSource:0}: Error finding container 7521e434794ff9fc02662ad894fe4afecdf6c33800c996d7314cf501f7764319: Status 404 returned error can't find the container with id 7521e434794ff9fc02662ad894fe4afecdf6c33800c996d7314cf501f7764319 Nov 03 18:38:08.197299 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:08.197269 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5c02ff_a13f_4d45_bb84_df74ec8fd86f.slice/crio-478ce0b62f04217c66b36c69f25ea4e49ab74ce5a224dc33d65405e1a4da1ef7 WatchSource:0}: Error finding container 478ce0b62f04217c66b36c69f25ea4e49ab74ce5a224dc33d65405e1a4da1ef7: Status 404 returned error can't find the container with id 478ce0b62f04217c66b36c69f25ea4e49ab74ce5a224dc33d65405e1a4da1ef7 Nov 03 18:38:08.198122 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:08.198094 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1827e8_4e13_48b4_a80f_4b2ef3d3d2e9.slice/crio-ea4214641c595eba67ae9c04a6a4641246fd63cb536c0778cd1628bfded27b43 WatchSource:0}: Error finding container ea4214641c595eba67ae9c04a6a4641246fd63cb536c0778cd1628bfded27b43: Status 404 returned error can't find the container with id ea4214641c595eba67ae9c04a6a4641246fd63cb536c0778cd1628bfded27b43 Nov 03 18:38:08.242392 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.242331 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf"] Nov 03 18:38:08.244190 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:08.244160 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16fa79d_4ad1_4fb4_b60b_915f0bbcb677.slice/crio-dc8ea6c5a6debc1b587ff0e929da8dc92448d8e1613ef3bc338c122ed7b31bf3 WatchSource:0}: Error finding container dc8ea6c5a6debc1b587ff0e929da8dc92448d8e1613ef3bc338c122ed7b31bf3: Status 404 returned error can't find the container with id dc8ea6c5a6debc1b587ff0e929da8dc92448d8e1613ef3bc338c122ed7b31bf3 Nov 03 18:38:08.245913 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.245891 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7577979dbf-qm2k5"] Nov 03 18:38:08.246365 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.246341 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66b9b978fb-5lvgf"] Nov 03 18:38:08.250277 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:08.250256 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aae4dec_41f5_4cc5_809d_cc496fdb1d77.slice/crio-fab84dbab44a3fc124618fdf42835a50874d829f6dc64a848729406ab794bf9c WatchSource:0}: Error finding container fab84dbab44a3fc124618fdf42835a50874d829f6dc64a848729406ab794bf9c: Status 404 returned error can't find the container with id fab84dbab44a3fc124618fdf42835a50874d829f6dc64a848729406ab794bf9c Nov 03 18:38:08.293450 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.293425 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wt7n9"] Nov 03 18:38:08.293970 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.293954 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74945cfd6f-j4qs2"] Nov 03 18:38:08.890692 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.890659 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" event={"ID":"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677","Type":"ContainerStarted","Data":"17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc"} Nov 03 18:38:08.891153 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.890728 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" event={"ID":"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677","Type":"ContainerStarted","Data":"dc8ea6c5a6debc1b587ff0e929da8dc92448d8e1613ef3bc338c122ed7b31bf3"} Nov 03 18:38:08.891153 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.890871 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:38:08.892333 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.892311 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" event={"ID":"8aae4dec-41f5-4cc5-809d-cc496fdb1d77","Type":"ContainerStarted","Data":"8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6"} Nov 03 18:38:08.892418 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.892347 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" event={"ID":"8aae4dec-41f5-4cc5-809d-cc496fdb1d77","Type":"ContainerStarted","Data":"fab84dbab44a3fc124618fdf42835a50874d829f6dc64a848729406ab794bf9c"} Nov 03 18:38:08.892418 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.892372 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:38:08.893566 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.893538 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt7n9" event={"ID":"edef417a-da49-4ca8-a744-0a3c7569546f","Type":"ContainerStarted","Data":"bbc324557b43aa23e602f6f57265512dce7da98907b9ed42cd9ee64149939157"} Nov 03 18:38:08.894746 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.894719 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" event={"ID":"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f","Type":"ContainerStarted","Data":"478ce0b62f04217c66b36c69f25ea4e49ab74ce5a224dc33d65405e1a4da1ef7"} Nov 03 18:38:08.895745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.895720 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5g7x" event={"ID":"c3e13ce6-21d8-4d9c-b589-46699c28b856","Type":"ContainerStarted","Data":"7521e434794ff9fc02662ad894fe4afecdf6c33800c996d7314cf501f7764319"} Nov 03 18:38:08.897070 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.897053 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7577979dbf-qm2k5" event={"ID":"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9","Type":"ContainerStarted","Data":"4ccc2699f82b3319e0185517458373ff39e0c80eb70775b7e0890fcf6f34d5d8"} Nov 03 18:38:08.897144 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:08.897082 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7577979dbf-qm2k5" event={"ID":"0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9","Type":"ContainerStarted","Data":"ea4214641c595eba67ae9c04a6a4641246fd63cb536c0778cd1628bfded27b43"} Nov 03 18:38:09.418606 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:09.418514 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66b9b978fb-5lvgf"] Nov 03 18:38:09.695061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:09.695024 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:09.698756 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:09.698496 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:09.901781 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:09.901750 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:09.903070 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:09.903049 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7577979dbf-qm2k5" Nov 03 18:38:10.906952 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:10.906905 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5g7x" event={"ID":"c3e13ce6-21d8-4d9c-b589-46699c28b856","Type":"ContainerStarted","Data":"ef100973b9d7de9ae61ed588dcbc70772ebeab5b4f8cdd2b6237d9d323770803"} Nov 03 18:38:10.907416 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:10.907063 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5g7x" event={"ID":"c3e13ce6-21d8-4d9c-b589-46699c28b856","Type":"ContainerStarted","Data":"d09afc08865699febf60dc4efa21d77540a39770a2ec7af3065895e60f57783e"} Nov 03 18:38:11.912875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:11.912795 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-75cdf" event={"ID":"fd5c02ff-a13f-4d45-bb84-df74ec8fd86f","Type":"ContainerStarted","Data":"16c1119f1143f511e031c374a41c0c7a11e1b1f45d95602b9aaf0c80d4a86a04"} Nov 03 18:38:11.914814 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:11.914787 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt7n9" event={"ID":"edef417a-da49-4ca8-a744-0a3c7569546f","Type":"ContainerStarted","Data":"77b4ba2b27e146c60927c85e8f976307167253d1a4d9ba9b7de50c7be7fa67fb"} Nov 03 18:38:11.914933 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:11.914832 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt7n9" event={"ID":"edef417a-da49-4ca8-a744-0a3c7569546f","Type":"ContainerStarted","Data":"7c7b5e2adb62983eab53b7937d5e2c7df509197c6e3796eef9fe06bf26d48edd"} Nov 03 18:38:11.915074 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:11.915057 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wt7n9" Nov 03 18:38:11.973632 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:11.973605 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92q5" Nov 03 18:38:18.895797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:18.895765 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nl6fg" Nov 03 18:38:19.534284 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:19.534255 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6bn8h" Nov 03 18:38:21.920379 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:21.920349 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wt7n9" Nov 03 18:38:24.746820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:24.746791 4948 scope.go:122] "RemoveContainer" containerID="a02ae980477e7bdb3d99cea2143d455de5371f7e6bf1c5d65955d2076739b985" Nov 03 18:38:24.962604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:24.962581 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:38:24.962777 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:24.962645 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" event={"ID":"8f64f0eb-fed7-496b-92a6-84601b0bb544","Type":"ContainerStarted","Data":"9792ac348b01957ec6f2bfdf0ab859ae2f5e659c51f02ed71b796ef1da0e0395"} Nov 03 18:38:24.962949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:24.962931 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:38:24.963899 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:24.963880 4948 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-v5gvn container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Nov 03 18:38:24.964009 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:24.963921 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Nov 03 18:38:25.537517 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.537479 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gv75s"] Nov 03 18:38:25.557333 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.557301 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.628134 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.628098 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 18:38:25.633606 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633581 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-wtmp\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.633742 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633612 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.633742 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633634 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-tls\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.633742 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633723 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-root\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.633868 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633786 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.633868 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633819 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-sys\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.633868 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633844 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-textfile\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.634004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633883 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2bf8da-0ed4-4795-b756-e9955bce76a8-metrics-client-ca\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.634004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.633947 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vvk8\" (UniqueName: \"kubernetes.io/projected/af2bf8da-0ed4-4795-b756-e9955bce76a8-kube-api-access-6vvk8\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.661995 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.661971 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nx4fl\"" Nov 03 18:38:25.661995 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.661975 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 18:38:25.681279 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.681253 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 18:38:25.681279 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.681256 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 18:38:25.734597 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734562 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-sys\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.734797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734609 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-textfile\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.734797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734631 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2bf8da-0ed4-4795-b756-e9955bce76a8-metrics-client-ca\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.734797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734657 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vvk8\" (UniqueName: \"kubernetes.io/projected/af2bf8da-0ed4-4795-b756-e9955bce76a8-kube-api-access-6vvk8\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.734797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734675 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-sys\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.734797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734724 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-wtmp\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.734797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734749 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.734797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734783 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-tls\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.735158 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734838 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-root\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.735158 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734868 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.735158 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734902 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-wtmp\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.735158 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734960 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af2bf8da-0ed4-4795-b756-e9955bce76a8-root\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.735158 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.734976 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-textfile\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.735368 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.735328 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.735368 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.735352 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2bf8da-0ed4-4795-b756-e9955bce76a8-metrics-client-ca\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.737165 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.737142 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-tls\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.737165 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.737152 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2bf8da-0ed4-4795-b756-e9955bce76a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.797138 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.797084 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vvk8\" (UniqueName: \"kubernetes.io/projected/af2bf8da-0ed4-4795-b756-e9955bce76a8-kube-api-access-6vvk8\") pod \"node-exporter-gv75s\" (UID: \"af2bf8da-0ed4-4795-b756-e9955bce76a8\") " pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.909592 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.909554 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gv75s" Nov 03 18:38:25.938674 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:38:25.938646 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2bf8da_0ed4_4795_b756_e9955bce76a8.slice/crio-22b4b202b666ac261d2e552f4d4966af82b7007835ee9a4827da9601d8c1a1bc WatchSource:0}: Error finding container 22b4b202b666ac261d2e552f4d4966af82b7007835ee9a4827da9601d8c1a1bc: Status 404 returned error can't find the container with id 22b4b202b666ac261d2e552f4d4966af82b7007835ee9a4827da9601d8c1a1bc Nov 03 18:38:25.967219 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.967189 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv75s" event={"ID":"af2bf8da-0ed4-4795-b756-e9955bce76a8","Type":"ContainerStarted","Data":"22b4b202b666ac261d2e552f4d4966af82b7007835ee9a4827da9601d8c1a1bc"} Nov 03 18:38:25.971187 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:25.971167 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" Nov 03 18:38:27.712062 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:27.712031 4948 patch_prober.go:29] interesting pod/image-registry-74945cfd6f-j4qs2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 18:38:27.712505 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:27.712088 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" podUID="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:38:27.978464 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:27.978383 4948 generic.go:356] "Generic (PLEG): container finished" podID="af2bf8da-0ed4-4795-b756-e9955bce76a8" containerID="8f4b58105581235410486263d19d3f1fc1043a1e8bc1721f9939f0aaabfaf9ba" exitCode=0 Nov 03 18:38:27.978464 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:27.978445 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv75s" event={"ID":"af2bf8da-0ed4-4795-b756-e9955bce76a8","Type":"ContainerDied","Data":"8f4b58105581235410486263d19d3f1fc1043a1e8bc1721f9939f0aaabfaf9ba"} Nov 03 18:38:28.984206 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:28.984168 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv75s" event={"ID":"af2bf8da-0ed4-4795-b756-e9955bce76a8","Type":"ContainerStarted","Data":"e29f46cf0aeba138b61e7488cabc2940eb38e38700692c34769022a509e84fcf"} Nov 03 18:38:28.984599 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:28.984235 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv75s" event={"ID":"af2bf8da-0ed4-4795-b756-e9955bce76a8","Type":"ContainerStarted","Data":"07eb347edca02c0f07514b886daa33e18ee292b2e384fb0e3d2870c5ede7786b"} Nov 03 18:38:29.909103 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:29.909074 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:38:30.034029 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:30.033978 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gv75s" podStartSLOduration=3.530993891 podStartE2EDuration="5.033964401s" totalImagesPullingTime="1.436253572s" totalInitContainerRuntime="66.716938ms" isStatefulPod=true podCreationTimestamp="2026-11-03 18:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:29.157710468 +0000 UTC m=+38.640706405" watchObservedRunningTime="2026-11-03 18:38:30.033964401 +0000 UTC m=+39.516960294" Nov 03 18:38:30.912809 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:30.912739 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:38:32.314850 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:32.314812 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74945cfd6f-j4qs2"] Nov 03 18:38:32.793028 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:38:32.792993 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:38:35.926203 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:35.926159 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" podUID="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" containerName="registry" containerID="cri-o://17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc" gracePeriod=30 Nov 03 18:38:36.159952 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.159929 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:38:36.217069 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217044 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217075 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217115 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217136 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217221 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217439 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217413 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217481 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217543 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217515 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\") pod \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\" (UID: \"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677\") " Nov 03 18:38:36.217796 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.217736 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:38:36.218130 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.218094 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:38:36.220193 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.220165 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:36.220296 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.220200 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:36.220296 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.220183 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:38:36.220296 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.220274 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "kube-api-access-7vqbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:36.220296 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.220284 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:38:36.224536 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.224465 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted" pod "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" (UID: "d16fa79d-4ad1-4fb4-b60b-915f0bbcb677"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:38:36.318994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.318963 4948 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-bound-sa-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:36.318994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.318990 4948 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-ca-trust-extracted\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:36.318994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.318999 4948 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-certificates\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:36.319218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.319009 4948 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-installation-pull-secrets\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:36.319218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.319019 4948 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-image-registry-private-configuration\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:36.319218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.319030 4948 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-trusted-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:36.319218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.319039 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqbv\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-kube-api-access-7vqbv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:36.319218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:36.319048 4948 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677-registry-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:37.015738 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.015641 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" Nov 03 18:38:37.015738 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.015661 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" event={"ID":"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677","Type":"ContainerDied","Data":"17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc"} Nov 03 18:38:37.015738 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.015601 4948 generic.go:356] "Generic (PLEG): container finished" podID="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" containerID="17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc" exitCode=0 Nov 03 18:38:37.015738 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.015737 4948 scope.go:122] "RemoveContainer" containerID="17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc" Nov 03 18:38:37.016246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.015803 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b9b978fb-5lvgf" event={"ID":"d16fa79d-4ad1-4fb4-b60b-915f0bbcb677","Type":"ContainerDied","Data":"dc8ea6c5a6debc1b587ff0e929da8dc92448d8e1613ef3bc338c122ed7b31bf3"} Nov 03 18:38:37.024356 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.024338 4948 scope.go:122] "RemoveContainer" containerID="17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc" Nov 03 18:38:37.024621 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:38:37.024602 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc\": container with ID starting with 17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc not found: ID does not exist" containerID="17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc" Nov 03 18:38:37.024671 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.024629 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc"} err="failed to get container status \"17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc\": rpc error: code = NotFound desc = could not find container \"17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc\": container with ID starting with 17baf82ffe9e22958069bb73f63d5927dbd5913665748b42a14a158c59ffa1bc not found: ID does not exist" Nov 03 18:38:37.238266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.238227 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66b9b978fb-5lvgf"] Nov 03 18:38:37.289123 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:37.289045 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66b9b978fb-5lvgf"] Nov 03 18:38:38.750857 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:38.750824 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" path="/var/lib/kubelet/pods/d16fa79d-4ad1-4fb4-b60b-915f0bbcb677/volumes" Nov 03 18:38:57.334330 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.334282 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" podUID="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" containerName="registry" containerID="cri-o://8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6" gracePeriod=30 Nov 03 18:38:57.594600 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.594551 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:38:57.692462 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692431 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.692619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692473 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.692619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692503 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.692619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692538 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.692619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692565 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.692619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692592 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.692900 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692622 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.692900 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692669 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\") pod \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\" (UID: \"8aae4dec-41f5-4cc5-809d-cc496fdb1d77\") " Nov 03 18:38:57.693015 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.692948 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:38:57.693073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.693052 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:38:57.695339 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.695297 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "kube-api-access-2dzzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:57.695339 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.695305 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:38:57.695558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.695428 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:38:57.695558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.695503 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:57.695558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.695503 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:57.699836 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.699812 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted" pod "8aae4dec-41f5-4cc5-809d-cc496fdb1d77" (UID: "8aae4dec-41f5-4cc5-809d-cc496fdb1d77"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:38:57.793359 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793330 4948 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-certificates\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:57.793359 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793356 4948 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-registry-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:57.793359 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793366 4948 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-bound-sa-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:57.793558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793376 4948 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-image-registry-private-configuration\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:57.793558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793386 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzzp\" (UniqueName: \"kubernetes.io/projected/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-kube-api-access-2dzzp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:57.793558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793395 4948 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-ca-trust-extracted\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:57.793558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793404 4948 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-trusted-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:57.793558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:57.793412 4948 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aae4dec-41f5-4cc5-809d-cc496fdb1d77-installation-pull-secrets\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:58.099145 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.099113 4948 generic.go:356] "Generic (PLEG): container finished" podID="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" containerID="8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6" exitCode=0 Nov 03 18:38:58.099327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.099166 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" Nov 03 18:38:58.099327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.099202 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" event={"ID":"8aae4dec-41f5-4cc5-809d-cc496fdb1d77","Type":"ContainerDied","Data":"8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6"} Nov 03 18:38:58.099327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.099253 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74945cfd6f-j4qs2" event={"ID":"8aae4dec-41f5-4cc5-809d-cc496fdb1d77","Type":"ContainerDied","Data":"fab84dbab44a3fc124618fdf42835a50874d829f6dc64a848729406ab794bf9c"} Nov 03 18:38:58.099327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.099272 4948 scope.go:122] "RemoveContainer" containerID="8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6" Nov 03 18:38:58.107643 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.107628 4948 scope.go:122] "RemoveContainer" containerID="8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6" Nov 03 18:38:58.107918 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:38:58.107894 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6\": container with ID starting with 8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6 not found: ID does not exist" containerID="8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6" Nov 03 18:38:58.108014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.107924 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6"} err="failed to get container status \"8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6\": rpc error: code = NotFound desc = could not find container \"8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6\": container with ID starting with 8d919340410813e5b6b05b04eaa7f86255db397f16e9b6d298c491843d0c11d6 not found: ID does not exist" Nov 03 18:38:58.272580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.272546 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74945cfd6f-j4qs2"] Nov 03 18:38:58.323667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.323637 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74945cfd6f-j4qs2"] Nov 03 18:38:58.750743 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:38:58.750692 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" path="/var/lib/kubelet/pods/8aae4dec-41f5-4cc5-809d-cc496fdb1d77/volumes" Nov 03 18:39:02.791151 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:39:02.791116 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:39:06.130232 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:06.130156 4948 generic.go:356] "Generic (PLEG): container finished" podID="289c8acb-0083-4ec9-ab45-d5e94d47c490" containerID="cef5d39135df21ca14b30be086896158507c77bbfa4aa7550843b9ad419b9292" exitCode=0 Nov 03 18:39:06.130668 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:06.130231 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" event={"ID":"289c8acb-0083-4ec9-ab45-d5e94d47c490","Type":"ContainerDied","Data":"cef5d39135df21ca14b30be086896158507c77bbfa4aa7550843b9ad419b9292"} Nov 03 18:39:06.130668 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:06.130537 4948 scope.go:122] "RemoveContainer" containerID="cef5d39135df21ca14b30be086896158507c77bbfa4aa7550843b9ad419b9292" Nov 03 18:39:06.131893 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:06.131870 4948 generic.go:356] "Generic (PLEG): container finished" podID="e50d63ba-d363-48f5-a720-880cd7c895d0" containerID="1c4c256c55842a6ec7f2b709e70a1a2bac8eb381599f5d17bbdcaf2e1f647630" exitCode=0 Nov 03 18:39:06.131984 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:06.131904 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" event={"ID":"e50d63ba-d363-48f5-a720-880cd7c895d0","Type":"ContainerDied","Data":"1c4c256c55842a6ec7f2b709e70a1a2bac8eb381599f5d17bbdcaf2e1f647630"} Nov 03 18:39:06.132145 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:06.132133 4948 scope.go:122] "RemoveContainer" containerID="1c4c256c55842a6ec7f2b709e70a1a2bac8eb381599f5d17bbdcaf2e1f647630" Nov 03 18:39:07.137630 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:07.137592 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbtph" event={"ID":"e50d63ba-d363-48f5-a720-880cd7c895d0","Type":"ContainerStarted","Data":"7845e53dbb1285c8a672ef7686f2e7386eb0cb3c17f2db98ec7dc176b7a1eb53"} Nov 03 18:39:07.139310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:07.139287 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-546xx" event={"ID":"289c8acb-0083-4ec9-ab45-d5e94d47c490","Type":"ContainerStarted","Data":"b73fa15176ec6b02a2bcce5b22d1703f651f8917d2e3b3294f444c46797a9f1e"} Nov 03 18:39:21.186446 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:21.186413 4948 generic.go:356] "Generic (PLEG): container finished" podID="10497765-1e63-4159-870c-ede32ad1b170" containerID="70bc35c3478ac05b60e28f836e0097fe7219e0cb0636dce79974f4c7d9b41c82" exitCode=0 Nov 03 18:39:21.186954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:21.186488 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqwhz" event={"ID":"10497765-1e63-4159-870c-ede32ad1b170","Type":"ContainerDied","Data":"70bc35c3478ac05b60e28f836e0097fe7219e0cb0636dce79974f4c7d9b41c82"} Nov 03 18:39:21.186954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:21.186888 4948 scope.go:122] "RemoveContainer" containerID="70bc35c3478ac05b60e28f836e0097fe7219e0cb0636dce79974f4c7d9b41c82" Nov 03 18:39:22.196216 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:39:22.196179 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqwhz" event={"ID":"10497765-1e63-4159-870c-ede32ad1b170","Type":"ContainerStarted","Data":"7441c8bfd302db0e95d3abf7db1674d52d058a43a9fb6e80aa0f476028c52cb1"} Nov 03 18:39:32.791416 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:39:32.791378 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:40:02.791844 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:40:02.791763 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:40:32.791263 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:40:32.791226 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:41:02.794541 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:41:02.794503 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:41:32.791747 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:41:32.791658 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:42:02.791710 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:42:02.791656 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:42:32.792445 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:42:32.792410 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:42:50.753440 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:42:50.753403 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:42:50.753998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:42:50.753949 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:42:50.759990 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:42:50.759969 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:42:50.760495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:42:50.760478 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:43:02.791150 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:43:02.791061 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:43:30.458321 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.458279 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm"] Nov 03 18:43:30.458968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.458780 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" containerName="registry" Nov 03 18:43:30.458968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.458799 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" containerName="registry" Nov 03 18:43:30.458968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.458827 4948 cpu_manager.go:414] "containerMap: removing container" podUID="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" containerName="registry" Nov 03 18:43:30.458968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.458835 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" containerName="registry" Nov 03 18:43:30.458968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.458908 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aae4dec-41f5-4cc5-809d-cc496fdb1d77" containerName="registry" Nov 03 18:43:30.458968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.458922 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16fa79d-4ad1-4fb4-b60b-915f0bbcb677" containerName="registry" Nov 03 18:43:30.462107 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.462084 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.568128 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.568094 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 18:43:30.568128 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.568119 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-gfjck\"" Nov 03 18:43:30.568317 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.568157 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 18:43:30.569108 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.569088 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm"] Nov 03 18:43:30.596106 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.596084 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pct\" (UniqueName: \"kubernetes.io/projected/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-kube-api-access-c6pct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.596196 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.596122 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.596234 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.596211 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.696734 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.696679 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.696898 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.696761 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.696898 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.696806 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pct\" (UniqueName: \"kubernetes.io/projected/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-kube-api-access-c6pct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.697080 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.697058 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.697119 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.697086 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.761182 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.761099 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pct\" (UniqueName: \"kubernetes.io/projected/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-kube-api-access-c6pct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:30.778234 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:30.778209 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:43:31.149904 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.149829 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm"] Nov 03 18:43:31.153597 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:31.153571 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05bb19f_0327_45d1_a2fb_31fb2284f7d2.slice/crio-d328cbeacab5bbe9cfd05e048f475d24ed4a1b51df49c1fcc5a4662b02fb44ed WatchSource:0}: Error finding container d328cbeacab5bbe9cfd05e048f475d24ed4a1b51df49c1fcc5a4662b02fb44ed: Status 404 returned error can't find the container with id d328cbeacab5bbe9cfd05e048f475d24ed4a1b51df49c1fcc5a4662b02fb44ed Nov 03 18:43:31.155858 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.155837 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:43:31.650744 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.650711 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b"] Nov 03 18:43:31.654058 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.654042 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.705202 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.705171 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzl7\" (UniqueName: \"kubernetes.io/projected/4000f1b0-85bd-4371-80d4-33973b11090d-kube-api-access-szzl7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.705319 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.705206 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.705319 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.705289 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.759618 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.759590 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-n8cqb\"" Nov 03 18:43:31.760622 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.760602 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b"] Nov 03 18:43:31.806212 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.806172 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.806360 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.806313 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzl7\" (UniqueName: \"kubernetes.io/projected/4000f1b0-85bd-4371-80d4-33973b11090d-kube-api-access-szzl7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.806360 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.806351 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.806634 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.806612 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.806728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.806630 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.869004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.868980 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzl7\" (UniqueName: \"kubernetes.io/projected/4000f1b0-85bd-4371-80d4-33973b11090d-kube-api-access-szzl7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.908062 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.908002 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p"] Nov 03 18:43:31.911335 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.911319 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:31.965546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.965518 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-mg52h\"" Nov 03 18:43:31.966861 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.966837 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p"] Nov 03 18:43:31.969770 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.969749 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:43:31.993927 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.993899 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55"] Nov 03 18:43:31.997636 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:31.997615 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.007932 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.007908 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.008059 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.007941 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgzd\" (UniqueName: \"kubernetes.io/projected/af86238a-5066-4a34-8cb8-dd2799317c7f-kube-api-access-jcgzd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.008059 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.007965 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.052742 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.052692 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55"] Nov 03 18:43:32.060391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.060361 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" event={"ID":"a05bb19f-0327-45d1-a2fb-31fb2284f7d2","Type":"ContainerStarted","Data":"d328cbeacab5bbe9cfd05e048f475d24ed4a1b51df49c1fcc5a4662b02fb44ed"} Nov 03 18:43:32.065578 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.065556 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-jnmts\"" Nov 03 18:43:32.093633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.093608 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7"] Nov 03 18:43:32.096954 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.096934 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.108966 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.108944 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.109055 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.108984 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.109055 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.109002 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgzd\" (UniqueName: \"kubernetes.io/projected/af86238a-5066-4a34-8cb8-dd2799317c7f-kube-api-access-jcgzd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.109126 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.109072 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.109160 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.109124 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.109196 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.109167 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5jwh\" (UniqueName: \"kubernetes.io/projected/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-kube-api-access-t5jwh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.109383 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.109362 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.109419 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.109411 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.151928 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.151900 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7"] Nov 03 18:43:32.165659 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.165607 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5l52z\"" Nov 03 18:43:32.170617 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.170591 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgzd\" (UniqueName: \"kubernetes.io/projected/af86238a-5066-4a34-8cb8-dd2799317c7f-kube-api-access-jcgzd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.203890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.203866 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c"] Nov 03 18:43:32.207299 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.207280 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.209811 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.209792 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.209909 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.209824 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7jqj\" (UniqueName: \"kubernetes.io/projected/e473e599-cc1f-4af8-81b0-cdf1b586239f-kube-api-access-p7jqj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.209909 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.209850 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5jwh\" (UniqueName: \"kubernetes.io/projected/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-kube-api-access-t5jwh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.209909 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.209875 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.210151 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.210125 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.210207 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.210166 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.210207 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.210192 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.210449 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.210434 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.227506 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.227488 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:43:32.263266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.263238 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c"] Nov 03 18:43:32.271309 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.271282 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9dczs\"" Nov 03 18:43:32.279561 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.279531 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5jwh\" (UniqueName: \"kubernetes.io/projected/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-kube-api-access-t5jwh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.311185 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311154 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.311302 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311211 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7jqj\" (UniqueName: \"kubernetes.io/projected/e473e599-cc1f-4af8-81b0-cdf1b586239f-kube-api-access-p7jqj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.311349 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311303 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.311406 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311346 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7r2\" (UniqueName: \"kubernetes.io/projected/3050177e-3803-4350-9428-cc2766b05a34-kube-api-access-7d7r2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.311406 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311382 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.311495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311413 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.311781 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311729 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.311781 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.311776 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.315109 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.315089 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:43:32.347573 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.347546 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b"] Nov 03 18:43:32.349216 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:32.349192 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4000f1b0_85bd_4371_80d4_33973b11090d.slice/crio-02aec2db3a72db1c41d8778ff5a5aaa7c8b4d05ea7f33adb50310a9aaf6a051f WatchSource:0}: Error finding container 02aec2db3a72db1c41d8778ff5a5aaa7c8b4d05ea7f33adb50310a9aaf6a051f: Status 404 returned error can't find the container with id 02aec2db3a72db1c41d8778ff5a5aaa7c8b4d05ea7f33adb50310a9aaf6a051f Nov 03 18:43:32.375577 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.375551 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7jqj\" (UniqueName: \"kubernetes.io/projected/e473e599-cc1f-4af8-81b0-cdf1b586239f-kube-api-access-p7jqj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.412381 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.412355 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.412505 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.412402 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7r2\" (UniqueName: \"kubernetes.io/projected/3050177e-3803-4350-9428-cc2766b05a34-kube-api-access-7d7r2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.412505 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.412430 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.412743 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.412722 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.412788 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.412754 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:43:32.412826 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.412795 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.475172 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.475142 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7r2\" (UniqueName: \"kubernetes.io/projected/3050177e-3803-4350-9428-cc2766b05a34-kube-api-access-7d7r2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.523528 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.523500 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:43:32.604889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.604858 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p"] Nov 03 18:43:32.606610 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:32.606581 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf86238a_5066_4a34_8cb8_dd2799317c7f.slice/crio-6071c31ad5e86f1cd1b7a7a5393963b11773104e93dc1fb0c93613e592121a39 WatchSource:0}: Error finding container 6071c31ad5e86f1cd1b7a7a5393963b11773104e93dc1fb0c93613e592121a39: Status 404 returned error can't find the container with id 6071c31ad5e86f1cd1b7a7a5393963b11773104e93dc1fb0c93613e592121a39 Nov 03 18:43:32.689251 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.689192 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55"] Nov 03 18:43:32.692015 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:32.691989 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cac2cbe_a7ab_4a76_bc6d_a419588b3cb8.slice/crio-5fe9f242256f3e60ac606f0c2a1e4ed5d1e6a2e4fab38b880278ad7cdf3fe1e8 WatchSource:0}: Error finding container 5fe9f242256f3e60ac606f0c2a1e4ed5d1e6a2e4fab38b880278ad7cdf3fe1e8: Status 404 returned error can't find the container with id 5fe9f242256f3e60ac606f0c2a1e4ed5d1e6a2e4fab38b880278ad7cdf3fe1e8 Nov 03 18:43:32.782680 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.782649 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7"] Nov 03 18:43:32.785770 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:32.785739 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode473e599_cc1f_4af8_81b0_cdf1b586239f.slice/crio-327dfba93d179bfda569e815a7eb1e6791b9611df10569b190caf4270f45f4cb WatchSource:0}: Error finding container 327dfba93d179bfda569e815a7eb1e6791b9611df10569b190caf4270f45f4cb: Status 404 returned error can't find the container with id 327dfba93d179bfda569e815a7eb1e6791b9611df10569b190caf4270f45f4cb Nov 03 18:43:32.795750 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:43:32.795721 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:43:32.804315 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.804294 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2"] Nov 03 18:43:32.809167 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.809149 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:32.862998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.862970 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-mvzwf\"" Nov 03 18:43:32.863166 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.863143 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2"] Nov 03 18:43:32.894569 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.894540 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c"] Nov 03 18:43:32.897100 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:32.897072 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3050177e_3803_4350_9428_cc2766b05a34.slice/crio-c496f654f4767b390a99e3eaacdeb33ffc5c2e52d4cdaac7bba873c10fdf9b7c WatchSource:0}: Error finding container c496f654f4767b390a99e3eaacdeb33ffc5c2e52d4cdaac7bba873c10fdf9b7c: Status 404 returned error can't find the container with id c496f654f4767b390a99e3eaacdeb33ffc5c2e52d4cdaac7bba873c10fdf9b7c Nov 03 18:43:32.916937 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.916910 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:32.917038 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.916975 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:32.917086 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:32.917041 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6b7\" (UniqueName: \"kubernetes.io/projected/6b8caa58-5a47-449a-b357-9c0a597f65be-kube-api-access-fn6b7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.018484 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.018402 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6b7\" (UniqueName: \"kubernetes.io/projected/6b8caa58-5a47-449a-b357-9c0a597f65be-kube-api-access-fn6b7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.018653 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.018485 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.018653 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.018540 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.018972 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.018942 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.019102 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.018981 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.067015 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.066975 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" event={"ID":"af86238a-5066-4a34-8cb8-dd2799317c7f","Type":"ContainerStarted","Data":"6071c31ad5e86f1cd1b7a7a5393963b11773104e93dc1fb0c93613e592121a39"} Nov 03 18:43:33.068436 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.068410 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" event={"ID":"3050177e-3803-4350-9428-cc2766b05a34","Type":"ContainerStarted","Data":"c496f654f4767b390a99e3eaacdeb33ffc5c2e52d4cdaac7bba873c10fdf9b7c"} Nov 03 18:43:33.069565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.069538 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" event={"ID":"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8","Type":"ContainerStarted","Data":"5fe9f242256f3e60ac606f0c2a1e4ed5d1e6a2e4fab38b880278ad7cdf3fe1e8"} Nov 03 18:43:33.070896 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.070856 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" event={"ID":"e473e599-cc1f-4af8-81b0-cdf1b586239f","Type":"ContainerStarted","Data":"327dfba93d179bfda569e815a7eb1e6791b9611df10569b190caf4270f45f4cb"} Nov 03 18:43:33.072226 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.072201 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" event={"ID":"4000f1b0-85bd-4371-80d4-33973b11090d","Type":"ContainerStarted","Data":"02aec2db3a72db1c41d8778ff5a5aaa7c8b4d05ea7f33adb50310a9aaf6a051f"} Nov 03 18:43:33.081640 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.081583 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6b7\" (UniqueName: \"kubernetes.io/projected/6b8caa58-5a47-449a-b357-9c0a597f65be-kube-api-access-fn6b7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.126048 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.126021 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:43:33.509151 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:33.509124 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2"] Nov 03 18:43:33.512092 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:33.512057 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8caa58_5a47_449a_b357_9c0a597f65be.slice/crio-a6bb2cf1d4fdb32b4b0189ec073c42c6d6de45692cc813c826f671958fbee62b WatchSource:0}: Error finding container a6bb2cf1d4fdb32b4b0189ec073c42c6d6de45692cc813c826f671958fbee62b: Status 404 returned error can't find the container with id a6bb2cf1d4fdb32b4b0189ec073c42c6d6de45692cc813c826f671958fbee62b Nov 03 18:43:34.083161 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:34.083121 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" event={"ID":"6b8caa58-5a47-449a-b357-9c0a597f65be","Type":"ContainerStarted","Data":"a6bb2cf1d4fdb32b4b0189ec073c42c6d6de45692cc813c826f671958fbee62b"} Nov 03 18:43:36.195814 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.195778 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx"] Nov 03 18:43:36.200175 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.200151 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.261114 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.261042 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx"] Nov 03 18:43:36.352783 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.352566 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.352783 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.352627 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.352783 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.352646 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qrkr\" (UniqueName: \"kubernetes.io/projected/e79d25f2-db4d-4bdb-b62f-c43743168455-kube-api-access-5qrkr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.453490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.453369 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.453490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.453472 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.453785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.453510 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qrkr\" (UniqueName: \"kubernetes.io/projected/e79d25f2-db4d-4bdb-b62f-c43743168455-kube-api-access-5qrkr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.454420 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.454388 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.454733 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.454694 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.537959 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.537900 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qrkr\" (UniqueName: \"kubernetes.io/projected/e79d25f2-db4d-4bdb-b62f-c43743168455-kube-api-access-5qrkr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:36.825665 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:36.825325 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:43:37.254063 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:37.253981 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx"] Nov 03 18:43:37.927980 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:37.927944 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc"] Nov 03 18:43:37.932928 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:37.932903 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:37.988626 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:37.988346 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 18:43:37.988626 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:37.988572 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h6x4t\"" Nov 03 18:43:38.040883 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.040664 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc"] Nov 03 18:43:38.071565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071502 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.071774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071632 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.071774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071672 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.071774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071716 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.071774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071751 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.072020 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071794 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.072020 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071893 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdw8\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-kube-api-access-lbdw8\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.072020 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.071962 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781701e-aa01-4167-84f2-0e2c7a7601dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.072176 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.072029 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.130582 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.130445 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" event={"ID":"e79d25f2-db4d-4bdb-b62f-c43743168455","Type":"ContainerStarted","Data":"21cdc49a06aad90f41426080017602dd42f138cea15e7b5c27e7dff1d0f32ecf"} Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173086 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781701e-aa01-4167-84f2-0e2c7a7601dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173158 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173196 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173225 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173260 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173290 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173323 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173355 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.173392 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdw8\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-kube-api-access-lbdw8\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.174750 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.174616 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.175568 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.175157 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.175568 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.175452 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.176024 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.175969 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.176173 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.176151 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781701e-aa01-4167-84f2-0e2c7a7601dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.187458 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.187435 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.195677 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.195632 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.237800 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.237768 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdw8\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-kube-api-access-lbdw8\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.250735 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.249269 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r"] Nov 03 18:43:38.252125 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.252098 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-vtbjc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.253723 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.253036 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.307683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.307593 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-znspf\"" Nov 03 18:43:38.316220 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.316070 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r"] Nov 03 18:43:38.375910 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.375873 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.375925 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.375953 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.375994 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.376021 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.376043 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376073 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.376066 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376368 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.376087 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.376368 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.376128 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5xg\" (UniqueName: \"kubernetes.io/projected/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-kube-api-access-ms5xg\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.463198 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.463107 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz"] Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477326 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477412 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477450 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477511 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477553 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477585 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477627 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477662 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.478726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.477741 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5xg\" (UniqueName: \"kubernetes.io/projected/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-kube-api-access-ms5xg\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.480295 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.479547 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.480295 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.479866 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.480740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.479873 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.481292 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.481268 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.481898 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.481874 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.482119 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.482091 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.493446 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.493381 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.497443 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.497421 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs"] Nov 03 18:43:38.498426 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.498401 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.506067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.506046 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.522606 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.522581 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz"] Nov 03 18:43:38.525227 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.525206 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn"] Nov 03 18:43:38.537627 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.537606 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.545710 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.545637 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5xg\" (UniqueName: \"kubernetes.io/projected/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-kube-api-access-ms5xg\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.551328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.551168 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:38.551931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.551906 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2e9bae-7530-4694-aa1a-4fcbfae723c4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r\" (UID: \"9c2e9bae-7530-4694-aa1a-4fcbfae723c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.563907 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.563880 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs"] Nov 03 18:43:38.564012 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.563946 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv"] Nov 03 18:43:38.571092 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.571016 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.578690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.578535 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:38.579918 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.579896 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcc7m\" (UniqueName: \"kubernetes.io/projected/e2b6beba-9422-4c15-a820-49a5c1e28a3f-kube-api-access-zcc7m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.580121 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.580101 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.580374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.580356 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8ls\" (UniqueName: \"kubernetes.io/projected/4d6aacab-a2ab-46bc-9587-7652af20c59d-kube-api-access-bx8ls\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.580584 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.580567 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.583484 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.583451 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.583661 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.583649 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.586540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.586523 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn"] Nov 03 18:43:38.620855 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.620693 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv"] Nov 03 18:43:38.685067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.684834 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.685067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.684878 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.685067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.684909 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnwp\" (UniqueName: \"kubernetes.io/projected/f1abe94a-0403-4391-a827-2e9a92f21c52-kube-api-access-zlnwp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.685067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.684936 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.685067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.684964 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.685067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.684995 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.685067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685035 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4vr\" (UniqueName: \"kubernetes.io/projected/7598b123-0778-40e9-9d5f-73ae0c7984eb-kube-api-access-cb4vr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.685540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685084 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcc7m\" (UniqueName: \"kubernetes.io/projected/e2b6beba-9422-4c15-a820-49a5c1e28a3f-kube-api-access-zcc7m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.685540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685103 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.685540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685121 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8ls\" (UniqueName: \"kubernetes.io/projected/4d6aacab-a2ab-46bc-9587-7652af20c59d-kube-api-access-bx8ls\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.685540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685143 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.685540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685157 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.685540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685518 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.685948 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.685822 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.686737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.686391 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.686737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.686450 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.752860 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.752782 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8ls\" (UniqueName: \"kubernetes.io/projected/4d6aacab-a2ab-46bc-9587-7652af20c59d-kube-api-access-bx8ls\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.753016 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.752956 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcc7m\" (UniqueName: \"kubernetes.io/projected/e2b6beba-9422-4c15-a820-49a5c1e28a3f-kube-api-access-zcc7m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.786608 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.786570 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.786808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.786629 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnwp\" (UniqueName: \"kubernetes.io/projected/f1abe94a-0403-4391-a827-2e9a92f21c52-kube-api-access-zlnwp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.786808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.786664 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.786808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.786687 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.786808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.786730 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.786808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.786790 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4vr\" (UniqueName: \"kubernetes.io/projected/7598b123-0778-40e9-9d5f-73ae0c7984eb-kube-api-access-cb4vr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.787538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.787512 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.788103 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.788078 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.788344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.788323 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.788579 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.788560 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.815138 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.815116 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:43:38.828779 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.828758 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:43:38.856236 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.856180 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnwp\" (UniqueName: \"kubernetes.io/projected/f1abe94a-0403-4391-a827-2e9a92f21c52-kube-api-access-zlnwp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.876975 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.876735 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:43:38.892604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.892484 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4vr\" (UniqueName: \"kubernetes.io/projected/7598b123-0778-40e9-9d5f-73ae0c7984eb-kube-api-access-cb4vr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.901029 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.900790 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:43:38.967156 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:38.967125 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc"] Nov 03 18:43:39.016132 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:39.016098 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r"] Nov 03 18:43:39.140737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:39.140606 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" event={"ID":"f781701e-aa01-4167-84f2-0e2c7a7601dc","Type":"ContainerStarted","Data":"f8ad043bec05eaeb4ccd88456d9624dec0a61c1f13e98c44c3d8fd27a911bd9d"} Nov 03 18:43:39.147914 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:39.147854 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" event={"ID":"9c2e9bae-7530-4694-aa1a-4fcbfae723c4","Type":"ContainerStarted","Data":"8aa3dfd5e662cae934622481711bdae89c3ae7c8f5642cce00477bd2f4b8fc9d"} Nov 03 18:43:39.236382 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:39.236267 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz"] Nov 03 18:43:39.268388 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:39.268359 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs"] Nov 03 18:43:39.276126 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:39.276070 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b6beba_9422_4c15_a820_49a5c1e28a3f.slice/crio-38ab19624fc272b4311f7320d9996ea409b042a2159518baeba25d3eaae1ce21 WatchSource:0}: Error finding container 38ab19624fc272b4311f7320d9996ea409b042a2159518baeba25d3eaae1ce21: Status 404 returned error can't find the container with id 38ab19624fc272b4311f7320d9996ea409b042a2159518baeba25d3eaae1ce21 Nov 03 18:43:39.319542 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:39.319504 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn"] Nov 03 18:43:39.324184 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:39.324152 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1abe94a_0403_4391_a827_2e9a92f21c52.slice/crio-872ee10fb475bb10c0acc9e564af8f675fd0e147538cb03e5144979b4b62564a WatchSource:0}: Error finding container 872ee10fb475bb10c0acc9e564af8f675fd0e147538cb03e5144979b4b62564a: Status 404 returned error can't find the container with id 872ee10fb475bb10c0acc9e564af8f675fd0e147538cb03e5144979b4b62564a Nov 03 18:43:39.404905 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:39.404734 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv"] Nov 03 18:43:39.416714 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:43:39.416657 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7598b123_0778_40e9_9d5f_73ae0c7984eb.slice/crio-a6a7a7076345a525b4a73d7c62e30f124496fece72b0bcaec6e67c3d2a94d18b WatchSource:0}: Error finding container a6a7a7076345a525b4a73d7c62e30f124496fece72b0bcaec6e67c3d2a94d18b: Status 404 returned error can't find the container with id a6a7a7076345a525b4a73d7c62e30f124496fece72b0bcaec6e67c3d2a94d18b Nov 03 18:43:40.168119 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:40.168080 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" event={"ID":"4d6aacab-a2ab-46bc-9587-7652af20c59d","Type":"ContainerStarted","Data":"0ce2737ef843e8d1d79f40006b480fe409b741093fab6d147b3677ff36d4e3b5"} Nov 03 18:43:40.173315 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:40.173282 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" event={"ID":"e2b6beba-9422-4c15-a820-49a5c1e28a3f","Type":"ContainerStarted","Data":"38ab19624fc272b4311f7320d9996ea409b042a2159518baeba25d3eaae1ce21"} Nov 03 18:43:40.184951 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:40.184924 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" event={"ID":"7598b123-0778-40e9-9d5f-73ae0c7984eb","Type":"ContainerStarted","Data":"a6a7a7076345a525b4a73d7c62e30f124496fece72b0bcaec6e67c3d2a94d18b"} Nov 03 18:43:40.190547 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:40.190517 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" event={"ID":"f1abe94a-0403-4391-a827-2e9a92f21c52","Type":"ContainerStarted","Data":"872ee10fb475bb10c0acc9e564af8f675fd0e147538cb03e5144979b4b62564a"} Nov 03 18:43:52.746779 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:52.746741 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:43:52.747251 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:52.746835 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:43:52.747251 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:52.746886 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:43:52.749604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:52.749576 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:43:52.749719 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:52.749644 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:43:52.749767 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:52.749686 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:43:53.388236 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.388135 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" event={"ID":"f781701e-aa01-4167-84f2-0e2c7a7601dc","Type":"ContainerStarted","Data":"4a22ef7fd92ba9b0a12c40408a99ebce65cc489bc9ebf24d8d43afbd65ad3c5b"} Nov 03 18:43:53.396966 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.396927 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" event={"ID":"9c2e9bae-7530-4694-aa1a-4fcbfae723c4","Type":"ContainerStarted","Data":"1008bacffd1b38907bbe957baf8a8a0f2569418b516b86ea1f377ac0bd47301d"} Nov 03 18:43:53.552116 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.552065 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:53.552356 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.552180 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:53.557924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.557901 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:53.568955 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.568905 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" podStartSLOduration=2.8201525849999998 podStartE2EDuration="16.568890441s" totalImagesPullingTime="13.748737856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:53.566947347 +0000 UTC m=+363.049943261" watchObservedRunningTime="2026-11-03 18:43:53.568890441 +0000 UTC m=+363.051886352" Nov 03 18:43:53.579963 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.579824 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:53.579963 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.579898 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:53.585481 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.585456 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:53.947835 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:53.947762 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" podStartSLOduration=2.224234883 podStartE2EDuration="15.947747304s" totalImagesPullingTime="13.723512421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:53.823062379 +0000 UTC m=+363.306058294" watchObservedRunningTime="2026-11-03 18:43:53.947747304 +0000 UTC m=+363.430743218" Nov 03 18:43:54.417162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:54.417060 4948 generic.go:356] "Generic (PLEG): container finished" podID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerID="36d5001f0b707a1dd983ed2edf7466aa391fd74b00c1969286fd9f6d7efec8b0" exitCode=0 Nov 03 18:43:54.417621 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:54.417588 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" event={"ID":"a05bb19f-0327-45d1-a2fb-31fb2284f7d2","Type":"ContainerDied","Data":"36d5001f0b707a1dd983ed2edf7466aa391fd74b00c1969286fd9f6d7efec8b0"} Nov 03 18:43:54.421059 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:54.420643 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r" Nov 03 18:43:54.421059 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:54.420979 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:43:55.425275 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:55.425238 4948 generic.go:356] "Generic (PLEG): container finished" podID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerID="81682c703cffdef8c9127e68bdf3c2e229b4f796d3c1e110ba41960dcc2591dc" exitCode=0 Nov 03 18:43:55.425663 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:55.425404 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" event={"ID":"e2b6beba-9422-4c15-a820-49a5c1e28a3f","Type":"ContainerDied","Data":"81682c703cffdef8c9127e68bdf3c2e229b4f796d3c1e110ba41960dcc2591dc"} Nov 03 18:43:56.433694 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:56.433666 4948 generic.go:356] "Generic (PLEG): container finished" podID="4000f1b0-85bd-4371-80d4-33973b11090d" containerID="e365978b911babbe281e1757b546892bb4372a22ee2ce096e6d7431c774a1e0e" exitCode=0 Nov 03 18:43:56.434114 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:56.433736 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" event={"ID":"4000f1b0-85bd-4371-80d4-33973b11090d","Type":"ContainerDied","Data":"e365978b911babbe281e1757b546892bb4372a22ee2ce096e6d7431c774a1e0e"} Nov 03 18:43:56.436356 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:56.436301 4948 generic.go:356] "Generic (PLEG): container finished" podID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerID="1af2e1dc5618b0c3035c2e09b6bb2b6dcfc46731de17ddefa7d454fcac0639c8" exitCode=0 Nov 03 18:43:56.436457 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:56.436426 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" event={"ID":"7598b123-0778-40e9-9d5f-73ae0c7984eb","Type":"ContainerDied","Data":"1af2e1dc5618b0c3035c2e09b6bb2b6dcfc46731de17ddefa7d454fcac0639c8"} Nov 03 18:43:56.438322 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:56.438296 4948 generic.go:356] "Generic (PLEG): container finished" podID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerID="dc506e9ba6213e0d525f53d298d1f6371e3d6cea19135298d0ee0ea88c0ecb1d" exitCode=0 Nov 03 18:43:56.439072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:56.438607 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" event={"ID":"f1abe94a-0403-4391-a827-2e9a92f21c52","Type":"ContainerDied","Data":"dc506e9ba6213e0d525f53d298d1f6371e3d6cea19135298d0ee0ea88c0ecb1d"} Nov 03 18:43:57.444644 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.444616 4948 generic.go:356] "Generic (PLEG): container finished" podID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerID="1c07e59860814d7b6df4d04b5721a15b48c46542391018d564b93c5fdb187cb4" exitCode=0 Nov 03 18:43:57.445126 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.444691 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" event={"ID":"a05bb19f-0327-45d1-a2fb-31fb2284f7d2","Type":"ContainerDied","Data":"1c07e59860814d7b6df4d04b5721a15b48c46542391018d564b93c5fdb187cb4"} Nov 03 18:43:57.446920 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.446815 4948 generic.go:356] "Generic (PLEG): container finished" podID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerID="050e17bf5536188a18ef59c6e4a7361696334d224498d0b97037356174f60c8f" exitCode=0 Nov 03 18:43:57.447077 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.446914 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" event={"ID":"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8","Type":"ContainerDied","Data":"050e17bf5536188a18ef59c6e4a7361696334d224498d0b97037356174f60c8f"} Nov 03 18:43:57.449187 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.449123 4948 generic.go:356] "Generic (PLEG): container finished" podID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerID="09bff5938260770859b8e2bb25b700563b98d9d5ccb9bc278bf78a370fc0c669" exitCode=0 Nov 03 18:43:57.449415 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.449396 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" event={"ID":"e473e599-cc1f-4af8-81b0-cdf1b586239f","Type":"ContainerDied","Data":"09bff5938260770859b8e2bb25b700563b98d9d5ccb9bc278bf78a370fc0c669"} Nov 03 18:43:57.454295 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.454268 4948 generic.go:356] "Generic (PLEG): container finished" podID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerID="587cc92aa589fbc4b0d0169be8b4837ebc9476e2e70d21cdab096911310e33ab" exitCode=0 Nov 03 18:43:57.454393 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.454365 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" event={"ID":"4d6aacab-a2ab-46bc-9587-7652af20c59d","Type":"ContainerDied","Data":"587cc92aa589fbc4b0d0169be8b4837ebc9476e2e70d21cdab096911310e33ab"} Nov 03 18:43:57.458084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.458061 4948 generic.go:356] "Generic (PLEG): container finished" podID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerID="05a0df9774e1aa86ec85dd0700c75e8c0dda57e8ac7a71d67f665367b52c04ce" exitCode=0 Nov 03 18:43:57.458186 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:57.458113 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" event={"ID":"e79d25f2-db4d-4bdb-b62f-c43743168455","Type":"ContainerDied","Data":"05a0df9774e1aa86ec85dd0700c75e8c0dda57e8ac7a71d67f665367b52c04ce"} Nov 03 18:43:58.466801 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.466766 4948 generic.go:356] "Generic (PLEG): container finished" podID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerID="cf63a71d7d46f9baac13214ebbc7f94c52c83c5cd2230409bb256dc7a3d13667" exitCode=0 Nov 03 18:43:58.467196 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.466861 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" event={"ID":"6b8caa58-5a47-449a-b357-9c0a597f65be","Type":"ContainerDied","Data":"cf63a71d7d46f9baac13214ebbc7f94c52c83c5cd2230409bb256dc7a3d13667"} Nov 03 18:43:58.471920 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.471891 4948 generic.go:356] "Generic (PLEG): container finished" podID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerID="1a68ca0983274b17a2ea1d0da851202154c34b6eac34453ffd8c3508791fdc2f" exitCode=0 Nov 03 18:43:58.472191 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.472157 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" event={"ID":"e2b6beba-9422-4c15-a820-49a5c1e28a3f","Type":"ContainerDied","Data":"1a68ca0983274b17a2ea1d0da851202154c34b6eac34453ffd8c3508791fdc2f"} Nov 03 18:43:58.475830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.475683 4948 generic.go:356] "Generic (PLEG): container finished" podID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerID="8a40ea4d069a45ea5c4409e0ed910890a61591a1a97f2abc288c418694f5df36" exitCode=0 Nov 03 18:43:58.475830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.475782 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" event={"ID":"af86238a-5066-4a34-8cb8-dd2799317c7f","Type":"ContainerDied","Data":"8a40ea4d069a45ea5c4409e0ed910890a61591a1a97f2abc288c418694f5df36"} Nov 03 18:43:58.477900 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.477879 4948 generic.go:356] "Generic (PLEG): container finished" podID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerID="d832c86e2d7ee35fdb460133ffdd8e8747dfa0fd9dfae8d14ec979512276c423" exitCode=0 Nov 03 18:43:58.478012 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.477965 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" event={"ID":"7598b123-0778-40e9-9d5f-73ae0c7984eb","Type":"ContainerDied","Data":"d832c86e2d7ee35fdb460133ffdd8e8747dfa0fd9dfae8d14ec979512276c423"} Nov 03 18:43:58.481896 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.481299 4948 generic.go:356] "Generic (PLEG): container finished" podID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerID="3e91b6e043a317a346f6ff5058db635368447a9bf435bba2da3f6b5c87cbf8dc" exitCode=0 Nov 03 18:43:58.481896 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.481411 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" event={"ID":"f1abe94a-0403-4391-a827-2e9a92f21c52","Type":"ContainerDied","Data":"3e91b6e043a317a346f6ff5058db635368447a9bf435bba2da3f6b5c87cbf8dc"} Nov 03 18:43:58.486586 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.486563 4948 generic.go:356] "Generic (PLEG): container finished" podID="3050177e-3803-4350-9428-cc2766b05a34" containerID="09eb4b612791462b0feebef82daa23a752225d2c5b593c53de57d48ed77f9e01" exitCode=0 Nov 03 18:43:58.486674 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.486592 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" event={"ID":"3050177e-3803-4350-9428-cc2766b05a34","Type":"ContainerDied","Data":"09eb4b612791462b0feebef82daa23a752225d2c5b593c53de57d48ed77f9e01"} Nov 03 18:43:58.490939 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.490918 4948 generic.go:356] "Generic (PLEG): container finished" podID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerID="d42aee24928eb64cf77a20c2f985f63ba3391001d70c45dc475da1afc7f9717a" exitCode=0 Nov 03 18:43:58.491033 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:58.490962 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" event={"ID":"e79d25f2-db4d-4bdb-b62f-c43743168455","Type":"ContainerDied","Data":"d42aee24928eb64cf77a20c2f985f63ba3391001d70c45dc475da1afc7f9717a"} Nov 03 18:43:59.525280 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.525240 4948 generic.go:356] "Generic (PLEG): container finished" podID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerID="74cc941991eff272e8dc482ef418ca33795ebc1a7dcd2593a1e4dcea58d15ba7" exitCode=0 Nov 03 18:43:59.525778 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.525360 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" event={"ID":"af86238a-5066-4a34-8cb8-dd2799317c7f","Type":"ContainerDied","Data":"74cc941991eff272e8dc482ef418ca33795ebc1a7dcd2593a1e4dcea58d15ba7"} Nov 03 18:43:59.529112 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.529084 4948 generic.go:356] "Generic (PLEG): container finished" podID="3050177e-3803-4350-9428-cc2766b05a34" containerID="4a850e188ea99f9505fc6e844867b2e6314faf7ba95f0aba5133258d9615bc0e" exitCode=0 Nov 03 18:43:59.529385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.529253 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" event={"ID":"3050177e-3803-4350-9428-cc2766b05a34","Type":"ContainerDied","Data":"4a850e188ea99f9505fc6e844867b2e6314faf7ba95f0aba5133258d9615bc0e"} Nov 03 18:43:59.534167 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.534133 4948 generic.go:356] "Generic (PLEG): container finished" podID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerID="e571f8bb7c04139792bafa7ed7e35c379d5560d246da5a72e2d3bd41b38016cf" exitCode=0 Nov 03 18:43:59.534448 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.534382 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" event={"ID":"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8","Type":"ContainerDied","Data":"e571f8bb7c04139792bafa7ed7e35c379d5560d246da5a72e2d3bd41b38016cf"} Nov 03 18:43:59.537163 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.537144 4948 generic.go:356] "Generic (PLEG): container finished" podID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerID="db7367001b58d8d0938fed4fda6d126b67ddc6b1c4c2ae6b9576c716644b4bc8" exitCode=0 Nov 03 18:43:59.537344 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.537218 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" event={"ID":"e473e599-cc1f-4af8-81b0-cdf1b586239f","Type":"ContainerDied","Data":"db7367001b58d8d0938fed4fda6d126b67ddc6b1c4c2ae6b9576c716644b4bc8"} Nov 03 18:43:59.543006 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.542340 4948 generic.go:356] "Generic (PLEG): container finished" podID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerID="9f48b5c509ed92414c686d1cf9a55c9feeb3bc0a1a1607410570a08bea98577f" exitCode=0 Nov 03 18:43:59.543006 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:43:59.542426 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" event={"ID":"4d6aacab-a2ab-46bc-9587-7652af20c59d","Type":"ContainerDied","Data":"9f48b5c509ed92414c686d1cf9a55c9feeb3bc0a1a1607410570a08bea98577f"} Nov 03 18:44:01.582281 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:01.582242 4948 generic.go:356] "Generic (PLEG): container finished" podID="4000f1b0-85bd-4371-80d4-33973b11090d" containerID="162c89aff18e85bfbc98cfb38d1380df78c5061924121d16d1efed0dd0c16739" exitCode=0 Nov 03 18:44:01.582780 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:01.582348 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" event={"ID":"4000f1b0-85bd-4371-80d4-33973b11090d","Type":"ContainerDied","Data":"162c89aff18e85bfbc98cfb38d1380df78c5061924121d16d1efed0dd0c16739"} Nov 03 18:44:02.811041 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:44:02.811002 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:44:07.689732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:07.689587 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" event={"ID":"6b8caa58-5a47-449a-b357-9c0a597f65be","Type":"ContainerDied","Data":"596537c6d23cd10ef8111b93fe102fa2e101d999def46828a4937d7097f76998"} Nov 03 18:44:07.689732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:07.689686 4948 generic.go:356] "Generic (PLEG): container finished" podID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerID="596537c6d23cd10ef8111b93fe102fa2e101d999def46828a4937d7097f76998" exitCode=0 Nov 03 18:44:21.895486 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:21.895435 4948 generic.go:356] "Generic (PLEG): container finished" podID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerID="3d4ff91e55a959da2ec0a6d68beb4066e0a9b80043685499456496e0dd15b0c9" exitCode=0 Nov 03 18:44:21.895995 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:21.895597 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" event={"ID":"f1abe94a-0403-4391-a827-2e9a92f21c52","Type":"ContainerDied","Data":"3d4ff91e55a959da2ec0a6d68beb4066e0a9b80043685499456496e0dd15b0c9"} Nov 03 18:44:22.917884 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:22.917321 4948 generic.go:356] "Generic (PLEG): container finished" podID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerID="5aaa6eb8388f8f7a6ad9c6b33d2a476988c95f8fd1bbd31b344956e4a662d07a" exitCode=0 Nov 03 18:44:22.917884 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:22.917451 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" event={"ID":"e2b6beba-9422-4c15-a820-49a5c1e28a3f","Type":"ContainerDied","Data":"5aaa6eb8388f8f7a6ad9c6b33d2a476988c95f8fd1bbd31b344956e4a662d07a"} Nov 03 18:44:22.923944 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:22.923913 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" event={"ID":"a05bb19f-0327-45d1-a2fb-31fb2284f7d2","Type":"ContainerStarted","Data":"554ef6cfb0dc112a1ff2e4f5b237f44ea023f1f702fd93d5f26b1ab6d61e8673"} Nov 03 18:44:23.163366 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.163246 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" podStartSLOduration=3.032809901 podStartE2EDuration="53.163225322s" totalImagesPullingTime="50.130415421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:23.157680199 +0000 UTC m=+392.640676112" watchObservedRunningTime="2026-11-03 18:44:23.163225322 +0000 UTC m=+392.646221237" Nov 03 18:44:23.186942 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.186601 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:44:23.302437 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.301582 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-util\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-util\") pod \"f1abe94a-0403-4391-a827-2e9a92f21c52\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " Nov 03 18:44:23.302437 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.301674 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1abe94a-0403-4391-a827-2e9a92f21c52-kube-api-access-zlnwp\" (UniqueName: \"kubernetes.io/projected/f1abe94a-0403-4391-a827-2e9a92f21c52-kube-api-access-zlnwp\") pod \"f1abe94a-0403-4391-a827-2e9a92f21c52\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " Nov 03 18:44:23.302437 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.301786 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-bundle\") pod \"f1abe94a-0403-4391-a827-2e9a92f21c52\" (UID: \"f1abe94a-0403-4391-a827-2e9a92f21c52\") " Nov 03 18:44:23.305298 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.305243 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-bundle" pod "f1abe94a-0403-4391-a827-2e9a92f21c52" (UID: "f1abe94a-0403-4391-a827-2e9a92f21c52"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:23.306887 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.306334 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abe94a-0403-4391-a827-2e9a92f21c52-kube-api-access-zlnwp" pod "f1abe94a-0403-4391-a827-2e9a92f21c52" (UID: "f1abe94a-0403-4391-a827-2e9a92f21c52"). InnerVolumeSpecName "kube-api-access-zlnwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:23.311072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.311048 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-util" pod "f1abe94a-0403-4391-a827-2e9a92f21c52" (UID: "f1abe94a-0403-4391-a827-2e9a92f21c52"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:23.403027 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.402996 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:23.403166 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.403032 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnwp\" (UniqueName: \"kubernetes.io/projected/f1abe94a-0403-4391-a827-2e9a92f21c52-kube-api-access-zlnwp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:23.403166 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.403047 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1abe94a-0403-4391-a827-2e9a92f21c52-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:23.933763 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.933728 4948 generic.go:356] "Generic (PLEG): container finished" podID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerID="efd014f759a8942d041d346482a6c8aae0fa3f976e1611f25fed5f2831ef62ec" exitCode=0 Nov 03 18:44:23.934162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.933837 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" event={"ID":"6b8caa58-5a47-449a-b357-9c0a597f65be","Type":"ContainerDied","Data":"efd014f759a8942d041d346482a6c8aae0fa3f976e1611f25fed5f2831ef62ec"} Nov 03 18:44:23.938498 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.938271 4948 generic.go:356] "Generic (PLEG): container finished" podID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerID="4240685bad10e5d14739c36643bc963d120712766b7395759150f9943f47339c" exitCode=0 Nov 03 18:44:23.938498 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.938344 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" event={"ID":"7598b123-0778-40e9-9d5f-73ae0c7984eb","Type":"ContainerDied","Data":"4240685bad10e5d14739c36643bc963d120712766b7395759150f9943f47339c"} Nov 03 18:44:23.944740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.941512 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" event={"ID":"f1abe94a-0403-4391-a827-2e9a92f21c52","Type":"ContainerDied","Data":"872ee10fb475bb10c0acc9e564af8f675fd0e147538cb03e5144979b4b62564a"} Nov 03 18:44:23.944740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.941551 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="872ee10fb475bb10c0acc9e564af8f675fd0e147538cb03e5144979b4b62564a" Nov 03 18:44:23.944740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.941617 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679ldn" Nov 03 18:44:23.948248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.947892 4948 generic.go:356] "Generic (PLEG): container finished" podID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerID="554ef6cfb0dc112a1ff2e4f5b237f44ea023f1f702fd93d5f26b1ab6d61e8673" exitCode=0 Nov 03 18:44:23.948248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:23.948019 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" event={"ID":"a05bb19f-0327-45d1-a2fb-31fb2284f7d2","Type":"ContainerDied","Data":"554ef6cfb0dc112a1ff2e4f5b237f44ea023f1f702fd93d5f26b1ab6d61e8673"} Nov 03 18:44:24.124007 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.123917 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:44:24.214485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.214456 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-util\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-util\") pod \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " Nov 03 18:44:24.214579 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.214503 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-bundle\") pod \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " Nov 03 18:44:24.214579 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.214556 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b6beba-9422-4c15-a820-49a5c1e28a3f-kube-api-access-zcc7m\" (UniqueName: \"kubernetes.io/projected/e2b6beba-9422-4c15-a820-49a5c1e28a3f-kube-api-access-zcc7m\") pod \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\" (UID: \"e2b6beba-9422-4c15-a820-49a5c1e28a3f\") " Nov 03 18:44:24.217034 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.217004 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-bundle" pod "e2b6beba-9422-4c15-a820-49a5c1e28a3f" (UID: "e2b6beba-9422-4c15-a820-49a5c1e28a3f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:24.217721 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.217657 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b6beba-9422-4c15-a820-49a5c1e28a3f-kube-api-access-zcc7m" pod "e2b6beba-9422-4c15-a820-49a5c1e28a3f" (UID: "e2b6beba-9422-4c15-a820-49a5c1e28a3f"). InnerVolumeSpecName "kube-api-access-zcc7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:24.228949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.228915 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-util" pod "e2b6beba-9422-4c15-a820-49a5c1e28a3f" (UID: "e2b6beba-9422-4c15-a820-49a5c1e28a3f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:24.316102 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.316026 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcc7m\" (UniqueName: \"kubernetes.io/projected/e2b6beba-9422-4c15-a820-49a5c1e28a3f-kube-api-access-zcc7m\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:24.316102 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.316055 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:24.316102 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.316070 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2b6beba-9422-4c15-a820-49a5c1e28a3f-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:24.956925 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.956897 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" Nov 03 18:44:24.957426 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.957352 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqhqs" event={"ID":"e2b6beba-9422-4c15-a820-49a5c1e28a3f","Type":"ContainerDied","Data":"38ab19624fc272b4311f7320d9996ea409b042a2159518baeba25d3eaae1ce21"} Nov 03 18:44:24.957426 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.957391 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ab19624fc272b4311f7320d9996ea409b042a2159518baeba25d3eaae1ce21" Nov 03 18:44:24.961147 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.961116 4948 generic.go:356] "Generic (PLEG): container finished" podID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerID="dde3ccfe73f2f385c2d7ec6abde06dbbf974ad83a76e72c6ff5fa5802875582a" exitCode=0 Nov 03 18:44:24.961296 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.961206 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" event={"ID":"af86238a-5066-4a34-8cb8-dd2799317c7f","Type":"ContainerDied","Data":"dde3ccfe73f2f385c2d7ec6abde06dbbf974ad83a76e72c6ff5fa5802875582a"} Nov 03 18:44:24.964169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.964141 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" event={"ID":"3050177e-3803-4350-9428-cc2766b05a34","Type":"ContainerStarted","Data":"f2702360772647fc6efcc8dfda28019e8806cbd0e6c886c88d11199708e7a819"} Nov 03 18:44:24.967461 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.967435 4948 generic.go:356] "Generic (PLEG): container finished" podID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerID="49b876fcbe6c2367005752285bef993574e0ff3c7afcc74e8fed881ae2328032" exitCode=0 Nov 03 18:44:24.967659 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.967640 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" event={"ID":"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8","Type":"ContainerDied","Data":"49b876fcbe6c2367005752285bef993574e0ff3c7afcc74e8fed881ae2328032"} Nov 03 18:44:24.970677 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.970571 4948 generic.go:356] "Generic (PLEG): container finished" podID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerID="9c80eb1bfb77b4410fab6efa0adf4b158ea34a7a212ea17e01e6e48556e0af7f" exitCode=0 Nov 03 18:44:24.970677 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.970649 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" event={"ID":"e473e599-cc1f-4af8-81b0-cdf1b586239f","Type":"ContainerDied","Data":"9c80eb1bfb77b4410fab6efa0adf4b158ea34a7a212ea17e01e6e48556e0af7f"} Nov 03 18:44:24.974255 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:24.973916 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" event={"ID":"e79d25f2-db4d-4bdb-b62f-c43743168455","Type":"ContainerStarted","Data":"e0deb4f17bbff7b007c20a0a3e014a246167b18513f1dca7602dd834d6bcc5b5"} Nov 03 18:44:25.137458 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.137432 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:44:25.196003 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.195978 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:44:25.208505 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.208482 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:44:25.261721 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.261641 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" podStartSLOduration=2.533706301 podStartE2EDuration="49.261624124s" totalImagesPullingTime="46.727917823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:25.140444143 +0000 UTC m=+394.623440057" watchObservedRunningTime="2026-11-03 18:44:25.261624124 +0000 UTC m=+394.744620038" Nov 03 18:44:25.325074 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325008 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-kube-api-access-c6pct\" (UniqueName: \"kubernetes.io/projected/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-kube-api-access-c6pct\") pod \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " Nov 03 18:44:25.325222 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325102 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-util\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-util\") pod \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " Nov 03 18:44:25.325222 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325132 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7598b123-0778-40e9-9d5f-73ae0c7984eb-kube-api-access-cb4vr\" (UniqueName: \"kubernetes.io/projected/7598b123-0778-40e9-9d5f-73ae0c7984eb-kube-api-access-cb4vr\") pod \"7598b123-0778-40e9-9d5f-73ae0c7984eb\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " Nov 03 18:44:25.325222 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325167 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-util\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-util\") pod \"6b8caa58-5a47-449a-b357-9c0a597f65be\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " Nov 03 18:44:25.325222 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325198 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-util\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-util\") pod \"7598b123-0778-40e9-9d5f-73ae0c7984eb\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " Nov 03 18:44:25.325429 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325269 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-bundle\") pod \"7598b123-0778-40e9-9d5f-73ae0c7984eb\" (UID: \"7598b123-0778-40e9-9d5f-73ae0c7984eb\") " Nov 03 18:44:25.325429 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325306 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-bundle\") pod \"6b8caa58-5a47-449a-b357-9c0a597f65be\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " Nov 03 18:44:25.325429 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325356 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8caa58-5a47-449a-b357-9c0a597f65be-kube-api-access-fn6b7\" (UniqueName: \"kubernetes.io/projected/6b8caa58-5a47-449a-b357-9c0a597f65be-kube-api-access-fn6b7\") pod \"6b8caa58-5a47-449a-b357-9c0a597f65be\" (UID: \"6b8caa58-5a47-449a-b357-9c0a597f65be\") " Nov 03 18:44:25.325429 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.325392 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-bundle\") pod \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\" (UID: \"a05bb19f-0327-45d1-a2fb-31fb2284f7d2\") " Nov 03 18:44:25.326178 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.326144 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-bundle" pod "a05bb19f-0327-45d1-a2fb-31fb2284f7d2" (UID: "a05bb19f-0327-45d1-a2fb-31fb2284f7d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:25.327571 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.327191 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-bundle" pod "7598b123-0778-40e9-9d5f-73ae0c7984eb" (UID: "7598b123-0778-40e9-9d5f-73ae0c7984eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:25.328459 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.328423 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-kube-api-access-c6pct" pod "a05bb19f-0327-45d1-a2fb-31fb2284f7d2" (UID: "a05bb19f-0327-45d1-a2fb-31fb2284f7d2"). InnerVolumeSpecName "kube-api-access-c6pct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:25.329377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.329270 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-bundle" pod "6b8caa58-5a47-449a-b357-9c0a597f65be" (UID: "6b8caa58-5a47-449a-b357-9c0a597f65be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:25.329825 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.329682 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8caa58-5a47-449a-b357-9c0a597f65be-kube-api-access-fn6b7" pod "6b8caa58-5a47-449a-b357-9c0a597f65be" (UID: "6b8caa58-5a47-449a-b357-9c0a597f65be"). InnerVolumeSpecName "kube-api-access-fn6b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:25.329825 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.329769 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7598b123-0778-40e9-9d5f-73ae0c7984eb-kube-api-access-cb4vr" pod "7598b123-0778-40e9-9d5f-73ae0c7984eb" (UID: "7598b123-0778-40e9-9d5f-73ae0c7984eb"). InnerVolumeSpecName "kube-api-access-cb4vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:25.333790 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.333768 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-util" pod "7598b123-0778-40e9-9d5f-73ae0c7984eb" (UID: "7598b123-0778-40e9-9d5f-73ae0c7984eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:25.335014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.334990 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-util" pod "a05bb19f-0327-45d1-a2fb-31fb2284f7d2" (UID: "a05bb19f-0327-45d1-a2fb-31fb2284f7d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:25.335203 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.335188 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-util" pod "6b8caa58-5a47-449a-b357-9c0a597f65be" (UID: "6b8caa58-5a47-449a-b357-9c0a597f65be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:25.380130 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.380069 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" podStartSLOduration=3.814802261 podStartE2EDuration="53.380052703s" totalImagesPullingTime="49.565250442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:25.379849387 +0000 UTC m=+394.862845308" watchObservedRunningTime="2026-11-03 18:44:25.380052703 +0000 UTC m=+394.863048617" Nov 03 18:44:25.426882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426846 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.426882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426878 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn6b7\" (UniqueName: \"kubernetes.io/projected/6b8caa58-5a47-449a-b357-9c0a597f65be-kube-api-access-fn6b7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.427067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426895 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.427067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426914 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6pct\" (UniqueName: \"kubernetes.io/projected/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-kube-api-access-c6pct\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.427067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426928 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a05bb19f-0327-45d1-a2fb-31fb2284f7d2-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.427067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426940 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4vr\" (UniqueName: \"kubernetes.io/projected/7598b123-0778-40e9-9d5f-73ae0c7984eb-kube-api-access-cb4vr\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.427067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426954 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b8caa58-5a47-449a-b357-9c0a597f65be-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.427067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426966 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.427067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.426974 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7598b123-0778-40e9-9d5f-73ae0c7984eb-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:25.981067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.981039 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" Nov 03 18:44:25.981067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.981043 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qc6hm" event={"ID":"a05bb19f-0327-45d1-a2fb-31fb2284f7d2","Type":"ContainerDied","Data":"d328cbeacab5bbe9cfd05e048f475d24ed4a1b51df49c1fcc5a4662b02fb44ed"} Nov 03 18:44:25.981495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.981099 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d328cbeacab5bbe9cfd05e048f475d24ed4a1b51df49c1fcc5a4662b02fb44ed" Nov 03 18:44:25.983518 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.983492 4948 generic.go:356] "Generic (PLEG): container finished" podID="3050177e-3803-4350-9428-cc2766b05a34" containerID="f2702360772647fc6efcc8dfda28019e8806cbd0e6c886c88d11199708e7a819" exitCode=0 Nov 03 18:44:25.983661 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.983575 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" event={"ID":"3050177e-3803-4350-9428-cc2766b05a34","Type":"ContainerDied","Data":"f2702360772647fc6efcc8dfda28019e8806cbd0e6c886c88d11199708e7a819"} Nov 03 18:44:25.985836 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.985809 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" event={"ID":"4000f1b0-85bd-4371-80d4-33973b11090d","Type":"ContainerStarted","Data":"470ecc862a8a6653daedc1cbffbbbbf14e54147a8375ef73e56a362705120b67"} Nov 03 18:44:25.990501 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.990476 4948 generic.go:356] "Generic (PLEG): container finished" podID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerID="72a5953fee628846fe3fc84790befe4e9a1c1d0a525c43545d310e3595581a43" exitCode=0 Nov 03 18:44:25.990594 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.990564 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" event={"ID":"4d6aacab-a2ab-46bc-9587-7652af20c59d","Type":"ContainerDied","Data":"72a5953fee628846fe3fc84790befe4e9a1c1d0a525c43545d310e3595581a43"} Nov 03 18:44:25.992676 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.992652 4948 generic.go:356] "Generic (PLEG): container finished" podID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerID="e0deb4f17bbff7b007c20a0a3e014a246167b18513f1dca7602dd834d6bcc5b5" exitCode=0 Nov 03 18:44:25.992816 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.992739 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" event={"ID":"e79d25f2-db4d-4bdb-b62f-c43743168455","Type":"ContainerDied","Data":"e0deb4f17bbff7b007c20a0a3e014a246167b18513f1dca7602dd834d6bcc5b5"} Nov 03 18:44:25.994558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.994543 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" Nov 03 18:44:25.994639 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.994559 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448jxw2" event={"ID":"6b8caa58-5a47-449a-b357-9c0a597f65be","Type":"ContainerDied","Data":"a6bb2cf1d4fdb32b4b0189ec073c42c6d6de45692cc813c826f671958fbee62b"} Nov 03 18:44:25.994639 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.994593 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6bb2cf1d4fdb32b4b0189ec073c42c6d6de45692cc813c826f671958fbee62b" Nov 03 18:44:25.996777 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.996751 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" event={"ID":"7598b123-0778-40e9-9d5f-73ae0c7984eb","Type":"ContainerDied","Data":"a6a7a7076345a525b4a73d7c62e30f124496fece72b0bcaec6e67c3d2a94d18b"} Nov 03 18:44:25.996885 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.996792 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6a7a7076345a525b4a73d7c62e30f124496fece72b0bcaec6e67c3d2a94d18b" Nov 03 18:44:25.996885 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:25.996795 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lgnjv" Nov 03 18:44:26.134406 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.134384 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:44:26.162248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.162215 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:44:26.169259 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.169241 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:44:26.232867 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.232796 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-kube-api-access-t5jwh\" (UniqueName: \"kubernetes.io/projected/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-kube-api-access-t5jwh\") pod \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " Nov 03 18:44:26.232867 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.232866 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-bundle\") pod \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " Nov 03 18:44:26.233083 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.232899 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-util\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-util\") pod \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\" (UID: \"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8\") " Nov 03 18:44:26.233378 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.233353 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-bundle" pod "2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" (UID: "2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:26.234940 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.234914 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-kube-api-access-t5jwh" pod "2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" (UID: "2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8"). InnerVolumeSpecName "kube-api-access-t5jwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:26.237528 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.237509 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-util" pod "2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" (UID: "2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:26.334112 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334086 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-bundle\") pod \"e473e599-cc1f-4af8-81b0-cdf1b586239f\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " Nov 03 18:44:26.334269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334149 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-util\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-util\") pod \"e473e599-cc1f-4af8-81b0-cdf1b586239f\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " Nov 03 18:44:26.334269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334190 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-bundle\") pod \"af86238a-5066-4a34-8cb8-dd2799317c7f\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " Nov 03 18:44:26.334269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334219 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e473e599-cc1f-4af8-81b0-cdf1b586239f-kube-api-access-p7jqj\" (UniqueName: \"kubernetes.io/projected/e473e599-cc1f-4af8-81b0-cdf1b586239f-kube-api-access-p7jqj\") pod \"e473e599-cc1f-4af8-81b0-cdf1b586239f\" (UID: \"e473e599-cc1f-4af8-81b0-cdf1b586239f\") " Nov 03 18:44:26.334269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334265 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-util\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-util\") pod \"af86238a-5066-4a34-8cb8-dd2799317c7f\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " Nov 03 18:44:26.334501 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334325 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af86238a-5066-4a34-8cb8-dd2799317c7f-kube-api-access-jcgzd\" (UniqueName: \"kubernetes.io/projected/af86238a-5066-4a34-8cb8-dd2799317c7f-kube-api-access-jcgzd\") pod \"af86238a-5066-4a34-8cb8-dd2799317c7f\" (UID: \"af86238a-5066-4a34-8cb8-dd2799317c7f\") " Nov 03 18:44:26.334589 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334569 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.334642 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334594 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.334642 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334608 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5jwh\" (UniqueName: \"kubernetes.io/projected/2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8-kube-api-access-t5jwh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.334777 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334742 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-bundle" pod "af86238a-5066-4a34-8cb8-dd2799317c7f" (UID: "af86238a-5066-4a34-8cb8-dd2799317c7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:26.334998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.334970 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-bundle" pod "e473e599-cc1f-4af8-81b0-cdf1b586239f" (UID: "e473e599-cc1f-4af8-81b0-cdf1b586239f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:26.336570 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.336539 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af86238a-5066-4a34-8cb8-dd2799317c7f-kube-api-access-jcgzd" pod "af86238a-5066-4a34-8cb8-dd2799317c7f" (UID: "af86238a-5066-4a34-8cb8-dd2799317c7f"). InnerVolumeSpecName "kube-api-access-jcgzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:26.336667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.336584 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e473e599-cc1f-4af8-81b0-cdf1b586239f-kube-api-access-p7jqj" pod "e473e599-cc1f-4af8-81b0-cdf1b586239f" (UID: "e473e599-cc1f-4af8-81b0-cdf1b586239f"). InnerVolumeSpecName "kube-api-access-p7jqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:26.342843 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.342816 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-util" pod "e473e599-cc1f-4af8-81b0-cdf1b586239f" (UID: "e473e599-cc1f-4af8-81b0-cdf1b586239f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:26.342952 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.342934 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-util" pod "af86238a-5066-4a34-8cb8-dd2799317c7f" (UID: "af86238a-5066-4a34-8cb8-dd2799317c7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:26.436042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.436004 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgzd\" (UniqueName: \"kubernetes.io/projected/af86238a-5066-4a34-8cb8-dd2799317c7f-kube-api-access-jcgzd\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.436042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.436039 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.436042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.436051 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e473e599-cc1f-4af8-81b0-cdf1b586239f-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.436267 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.436060 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.436267 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.436068 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7jqj\" (UniqueName: \"kubernetes.io/projected/e473e599-cc1f-4af8-81b0-cdf1b586239f-kube-api-access-p7jqj\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.436267 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.436076 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af86238a-5066-4a34-8cb8-dd2799317c7f-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:26.451747 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:26.451674 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" podStartSLOduration=3.52384356 podStartE2EDuration="55.451660395s" totalImagesPullingTime="51.927816835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:26.451528156 +0000 UTC m=+395.934524067" watchObservedRunningTime="2026-11-03 18:44:26.451660395 +0000 UTC m=+395.934656308" Nov 03 18:44:27.005010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.004979 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" Nov 03 18:44:27.005430 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.005023 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jtm5p" event={"ID":"af86238a-5066-4a34-8cb8-dd2799317c7f","Type":"ContainerDied","Data":"6071c31ad5e86f1cd1b7a7a5393963b11773104e93dc1fb0c93613e592121a39"} Nov 03 18:44:27.005430 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.005068 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6071c31ad5e86f1cd1b7a7a5393963b11773104e93dc1fb0c93613e592121a39" Nov 03 18:44:27.007150 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.007117 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" event={"ID":"2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8","Type":"ContainerDied","Data":"5fe9f242256f3e60ac606f0c2a1e4ed5d1e6a2e4fab38b880278ad7cdf3fe1e8"} Nov 03 18:44:27.007293 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.007154 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c77w55" Nov 03 18:44:27.007293 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.007171 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fe9f242256f3e60ac606f0c2a1e4ed5d1e6a2e4fab38b880278ad7cdf3fe1e8" Nov 03 18:44:27.009339 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.009287 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" event={"ID":"e473e599-cc1f-4af8-81b0-cdf1b586239f","Type":"ContainerDied","Data":"327dfba93d179bfda569e815a7eb1e6791b9611df10569b190caf4270f45f4cb"} Nov 03 18:44:27.009339 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.009328 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="327dfba93d179bfda569e815a7eb1e6791b9611df10569b190caf4270f45f4cb" Nov 03 18:44:27.009563 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.009543 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj4tg7" Nov 03 18:44:27.011548 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.011526 4948 generic.go:356] "Generic (PLEG): container finished" podID="4000f1b0-85bd-4371-80d4-33973b11090d" containerID="470ecc862a8a6653daedc1cbffbbbbf14e54147a8375ef73e56a362705120b67" exitCode=0 Nov 03 18:44:27.011641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.011609 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" event={"ID":"4000f1b0-85bd-4371-80d4-33973b11090d","Type":"ContainerDied","Data":"470ecc862a8a6653daedc1cbffbbbbf14e54147a8375ef73e56a362705120b67"} Nov 03 18:44:27.146480 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.146360 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:44:27.171005 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.170982 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:44:27.178732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.178692 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:44:27.242938 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.242911 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-util\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-util\") pod \"4d6aacab-a2ab-46bc-9587-7652af20c59d\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " Nov 03 18:44:27.243076 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.242951 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6aacab-a2ab-46bc-9587-7652af20c59d-kube-api-access-bx8ls\" (UniqueName: \"kubernetes.io/projected/4d6aacab-a2ab-46bc-9587-7652af20c59d-kube-api-access-bx8ls\") pod \"4d6aacab-a2ab-46bc-9587-7652af20c59d\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " Nov 03 18:44:27.243076 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.243013 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-bundle\") pod \"4d6aacab-a2ab-46bc-9587-7652af20c59d\" (UID: \"4d6aacab-a2ab-46bc-9587-7652af20c59d\") " Nov 03 18:44:27.243776 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.243746 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-bundle" pod "4d6aacab-a2ab-46bc-9587-7652af20c59d" (UID: "4d6aacab-a2ab-46bc-9587-7652af20c59d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:27.244988 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.244969 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6aacab-a2ab-46bc-9587-7652af20c59d-kube-api-access-bx8ls" pod "4d6aacab-a2ab-46bc-9587-7652af20c59d" (UID: "4d6aacab-a2ab-46bc-9587-7652af20c59d"). InnerVolumeSpecName "kube-api-access-bx8ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:27.248784 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.248764 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-util" pod "4d6aacab-a2ab-46bc-9587-7652af20c59d" (UID: "4d6aacab-a2ab-46bc-9587-7652af20c59d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:27.343534 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343507 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-util\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-util\") pod \"3050177e-3803-4350-9428-cc2766b05a34\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " Nov 03 18:44:27.343635 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343539 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-bundle\") pod \"e79d25f2-db4d-4bdb-b62f-c43743168455\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " Nov 03 18:44:27.343635 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343567 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3050177e-3803-4350-9428-cc2766b05a34-kube-api-access-7d7r2\" (UniqueName: \"kubernetes.io/projected/3050177e-3803-4350-9428-cc2766b05a34-kube-api-access-7d7r2\") pod \"3050177e-3803-4350-9428-cc2766b05a34\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " Nov 03 18:44:27.343635 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343614 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-util\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-util\") pod \"e79d25f2-db4d-4bdb-b62f-c43743168455\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " Nov 03 18:44:27.343808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343646 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-bundle\") pod \"3050177e-3803-4350-9428-cc2766b05a34\" (UID: \"3050177e-3803-4350-9428-cc2766b05a34\") " Nov 03 18:44:27.343808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343722 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79d25f2-db4d-4bdb-b62f-c43743168455-kube-api-access-5qrkr\" (UniqueName: \"kubernetes.io/projected/e79d25f2-db4d-4bdb-b62f-c43743168455-kube-api-access-5qrkr\") pod \"e79d25f2-db4d-4bdb-b62f-c43743168455\" (UID: \"e79d25f2-db4d-4bdb-b62f-c43743168455\") " Nov 03 18:44:27.343967 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343949 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.344038 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343972 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8ls\" (UniqueName: \"kubernetes.io/projected/4d6aacab-a2ab-46bc-9587-7652af20c59d-kube-api-access-bx8ls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.344038 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.343990 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6aacab-a2ab-46bc-9587-7652af20c59d-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.344222 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.344196 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-bundle" pod "3050177e-3803-4350-9428-cc2766b05a34" (UID: "3050177e-3803-4350-9428-cc2766b05a34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:27.344271 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.344205 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-bundle" pod "e79d25f2-db4d-4bdb-b62f-c43743168455" (UID: "e79d25f2-db4d-4bdb-b62f-c43743168455"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:27.345957 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.345933 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79d25f2-db4d-4bdb-b62f-c43743168455-kube-api-access-5qrkr" pod "e79d25f2-db4d-4bdb-b62f-c43743168455" (UID: "e79d25f2-db4d-4bdb-b62f-c43743168455"). InnerVolumeSpecName "kube-api-access-5qrkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:27.346226 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.346202 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3050177e-3803-4350-9428-cc2766b05a34-kube-api-access-7d7r2" pod "3050177e-3803-4350-9428-cc2766b05a34" (UID: "3050177e-3803-4350-9428-cc2766b05a34"). InnerVolumeSpecName "kube-api-access-7d7r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:27.350001 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.349979 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-util" pod "3050177e-3803-4350-9428-cc2766b05a34" (UID: "3050177e-3803-4350-9428-cc2766b05a34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:27.352896 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.352872 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-util" pod "e79d25f2-db4d-4bdb-b62f-c43743168455" (UID: "e79d25f2-db4d-4bdb-b62f-c43743168455"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:27.444830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.444798 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.444830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.444826 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.444830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.444836 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qrkr\" (UniqueName: \"kubernetes.io/projected/e79d25f2-db4d-4bdb-b62f-c43743168455-kube-api-access-5qrkr\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.445022 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.444844 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3050177e-3803-4350-9428-cc2766b05a34-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.445022 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.444853 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e79d25f2-db4d-4bdb-b62f-c43743168455-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:27.445022 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:27.444865 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7r2\" (UniqueName: \"kubernetes.io/projected/3050177e-3803-4350-9428-cc2766b05a34-kube-api-access-7d7r2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:28.018660 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.018624 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" event={"ID":"4d6aacab-a2ab-46bc-9587-7652af20c59d","Type":"ContainerDied","Data":"0ce2737ef843e8d1d79f40006b480fe409b741093fab6d147b3677ff36d4e3b5"} Nov 03 18:44:28.019168 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.018687 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce2737ef843e8d1d79f40006b480fe409b741093fab6d147b3677ff36d4e3b5" Nov 03 18:44:28.019168 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.018641 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a76nfz" Nov 03 18:44:28.021223 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.021190 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" event={"ID":"e79d25f2-db4d-4bdb-b62f-c43743168455","Type":"ContainerDied","Data":"21cdc49a06aad90f41426080017602dd42f138cea15e7b5c27e7dff1d0f32ecf"} Nov 03 18:44:28.021407 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.021384 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21cdc49a06aad90f41426080017602dd42f138cea15e7b5c27e7dff1d0f32ecf" Nov 03 18:44:28.021407 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.021208 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ddhhx" Nov 03 18:44:28.023375 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.023340 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" Nov 03 18:44:28.023462 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.023366 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75vf7c" event={"ID":"3050177e-3803-4350-9428-cc2766b05a34","Type":"ContainerDied","Data":"c496f654f4767b390a99e3eaacdeb33ffc5c2e52d4cdaac7bba873c10fdf9b7c"} Nov 03 18:44:28.023462 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.023409 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c496f654f4767b390a99e3eaacdeb33ffc5c2e52d4cdaac7bba873c10fdf9b7c" Nov 03 18:44:28.157746 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.157724 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:44:28.352021 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.351952 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-util\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-util\") pod \"4000f1b0-85bd-4371-80d4-33973b11090d\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " Nov 03 18:44:28.352021 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.352004 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4000f1b0-85bd-4371-80d4-33973b11090d-kube-api-access-szzl7\" (UniqueName: \"kubernetes.io/projected/4000f1b0-85bd-4371-80d4-33973b11090d-kube-api-access-szzl7\") pod \"4000f1b0-85bd-4371-80d4-33973b11090d\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " Nov 03 18:44:28.352194 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.352031 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-bundle\") pod \"4000f1b0-85bd-4371-80d4-33973b11090d\" (UID: \"4000f1b0-85bd-4371-80d4-33973b11090d\") " Nov 03 18:44:28.352694 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.352670 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-bundle" pod "4000f1b0-85bd-4371-80d4-33973b11090d" (UID: "4000f1b0-85bd-4371-80d4-33973b11090d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:28.354133 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.354103 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4000f1b0-85bd-4371-80d4-33973b11090d-kube-api-access-szzl7" pod "4000f1b0-85bd-4371-80d4-33973b11090d" (UID: "4000f1b0-85bd-4371-80d4-33973b11090d"). InnerVolumeSpecName "kube-api-access-szzl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:44:28.357325 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.357303 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-util" pod "4000f1b0-85bd-4371-80d4-33973b11090d" (UID: "4000f1b0-85bd-4371-80d4-33973b11090d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:44:28.452788 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.452755 4948 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-util\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:28.452788 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.452782 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzl7\" (UniqueName: \"kubernetes.io/projected/4000f1b0-85bd-4371-80d4-33973b11090d-kube-api-access-szzl7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:28.452788 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:28.452791 4948 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4000f1b0-85bd-4371-80d4-33973b11090d-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:44:29.030215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:29.030134 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" Nov 03 18:44:29.030215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:29.030150 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv7g8b" event={"ID":"4000f1b0-85bd-4371-80d4-33973b11090d","Type":"ContainerDied","Data":"02aec2db3a72db1c41d8778ff5a5aaa7c8b4d05ea7f33adb50310a9aaf6a051f"} Nov 03 18:44:29.030215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:29.030198 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02aec2db3a72db1c41d8778ff5a5aaa7c8b4d05ea7f33adb50310a9aaf6a051f" Nov 03 18:44:30.464558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.464520 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6"] Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.464988 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerName="extract" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465007 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerName="extract" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465024 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4000f1b0-85bd-4371-80d4-33973b11090d" containerName="util" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465031 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4000f1b0-85bd-4371-80d4-33973b11090d" containerName="util" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465041 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerName="util" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465049 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerName="util" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465057 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerName="extract" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465064 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerName="extract" Nov 03 18:44:30.465075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465073 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465080 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465096 4948 cpu_manager.go:414] "containerMap: removing container" podUID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465103 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465113 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465120 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465131 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465138 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465148 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3050177e-3803-4350-9428-cc2766b05a34" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465155 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3050177e-3803-4350-9428-cc2766b05a34" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465167 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3050177e-3803-4350-9428-cc2766b05a34" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465174 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3050177e-3803-4350-9428-cc2766b05a34" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465184 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465192 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465203 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3050177e-3803-4350-9428-cc2766b05a34" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465210 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3050177e-3803-4350-9428-cc2766b05a34" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465220 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4000f1b0-85bd-4371-80d4-33973b11090d" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465227 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4000f1b0-85bd-4371-80d4-33973b11090d" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465238 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465245 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465257 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465264 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465274 4948 cpu_manager.go:414] "containerMap: removing container" podUID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465282 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465294 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465300 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465309 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465315 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465325 4948 cpu_manager.go:414] "containerMap: removing container" podUID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465331 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465343 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465351 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465362 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465369 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465379 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465385 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465395 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465402 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465416 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465422 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465435 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465442 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465450 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465457 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerName="extract" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465472 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465479 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465494 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465501 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerName="util" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465511 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465518 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465527 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4000f1b0-85bd-4371-80d4-33973b11090d" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465538 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4000f1b0-85bd-4371-80d4-33973b11090d" containerName="pull" Nov 03 18:44:30.465524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465549 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerName="util" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465555 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerName="util" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465566 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465575 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465585 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465592 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465604 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465611 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465626 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerName="util" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465633 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerName="util" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465643 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465650 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerName="pull" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465753 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7598b123-0778-40e9-9d5f-73ae0c7984eb" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465768 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b6beba-9422-4c15-a820-49a5c1e28a3f" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465776 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cac2cbe-a7ab-4a76-bc6d-a419588b3cb8" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465786 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6aacab-a2ab-46bc-9587-7652af20c59d" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465795 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05bb19f-0327-45d1-a2fb-31fb2284f7d2" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465804 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8caa58-5a47-449a-b357-9c0a597f65be" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465812 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79d25f2-db4d-4bdb-b62f-c43743168455" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465819 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4000f1b0-85bd-4371-80d4-33973b11090d" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465828 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="af86238a-5066-4a34-8cb8-dd2799317c7f" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465836 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e473e599-cc1f-4af8-81b0-cdf1b586239f" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465849 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3050177e-3803-4350-9428-cc2766b05a34" containerName="extract" Nov 03 18:44:30.467523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.465858 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1abe94a-0403-4391-a827-2e9a92f21c52" containerName="extract" Nov 03 18:44:30.493859 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.493834 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.555581 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.555547 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 18:44:30.555742 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.555557 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 18:44:30.555742 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.555618 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 18:44:30.569135 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.569108 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6bb9\" (UniqueName: \"kubernetes.io/projected/9f958cc0-6cf3-4141-9a06-c175802788e7-kube-api-access-x6bb9\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.569249 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.569139 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.569249 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.569218 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f958cc0-6cf3-4141-9a06-c175802788e7-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.569249 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.569241 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.569342 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.569263 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.569342 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.569297 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f958cc0-6cf3-4141-9a06-c175802788e7-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.605062 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.605035 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 18:44:30.605179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.605081 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jdkcr\"" Nov 03 18:44:30.605179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.605111 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 18:44:30.607626 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.607608 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6"] Nov 03 18:44:30.669712 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.669669 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6bb9\" (UniqueName: \"kubernetes.io/projected/9f958cc0-6cf3-4141-9a06-c175802788e7-kube-api-access-x6bb9\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.669874 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.669718 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.669874 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.669750 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f958cc0-6cf3-4141-9a06-c175802788e7-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.669874 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.669767 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.669874 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.669784 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.670072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.669978 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f958cc0-6cf3-4141-9a06-c175802788e7-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.670322 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.670286 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f958cc0-6cf3-4141-9a06-c175802788e7-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.672284 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.672252 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f958cc0-6cf3-4141-9a06-c175802788e7-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.672284 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.672271 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.672414 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.672363 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.672459 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.672399 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f958cc0-6cf3-4141-9a06-c175802788e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.731967 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.731884 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6bb9\" (UniqueName: \"kubernetes.io/projected/9f958cc0-6cf3-4141-9a06-c175802788e7-kube-api-access-x6bb9\") pod \"opendatahub-operator-controller-manager-7b94cbc577-2nqz6\" (UID: \"9f958cc0-6cf3-4141-9a06-c175802788e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:30.840816 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:30.840770 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:31.445788 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:31.445762 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6"] Nov 03 18:44:31.447457 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:44:31.447430 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f958cc0_6cf3_4141_9a06_c175802788e7.slice/crio-ce0a92eece548f3a3a090dd83a1d241068a28afd8e091c660c67500fe05309fa WatchSource:0}: Error finding container ce0a92eece548f3a3a090dd83a1d241068a28afd8e091c660c67500fe05309fa: Status 404 returned error can't find the container with id ce0a92eece548f3a3a090dd83a1d241068a28afd8e091c660c67500fe05309fa Nov 03 18:44:32.046187 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:32.046153 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" event={"ID":"9f958cc0-6cf3-4141-9a06-c175802788e7","Type":"ContainerStarted","Data":"ce0a92eece548f3a3a090dd83a1d241068a28afd8e091c660c67500fe05309fa"} Nov 03 18:44:32.793336 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:44:32.793300 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:44:37.075330 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:37.075296 4948 generic.go:356] "Generic (PLEG): container finished" podID="9f958cc0-6cf3-4141-9a06-c175802788e7" containerID="d51dcef545cf430ffa24eed5cc450156cbfc90bdb5569fafeceb5a4a8a4d8f6b" exitCode=0 Nov 03 18:44:37.075662 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:37.075355 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" event={"ID":"9f958cc0-6cf3-4141-9a06-c175802788e7","Type":"ContainerDied","Data":"d51dcef545cf430ffa24eed5cc450156cbfc90bdb5569fafeceb5a4a8a4d8f6b"} Nov 03 18:44:38.081822 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:38.081788 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" event={"ID":"9f958cc0-6cf3-4141-9a06-c175802788e7","Type":"ContainerStarted","Data":"75fc609754225cab21c2f4150d436920939f224d93f32289b75565e2b0e7d1b7"} Nov 03 18:44:38.082178 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:38.081878 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:38.324303 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:38.324255 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" podStartSLOduration=1.954934285 podStartE2EDuration="8.324239215s" totalImagesPullingTime="5.919420931s" totalInitContainerRuntime="449.883999ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:38.273040395 +0000 UTC m=+407.756036308" watchObservedRunningTime="2026-11-03 18:44:38.324239215 +0000 UTC m=+407.807235129" Nov 03 18:44:46.120575 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.120542 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz"] Nov 03 18:44:46.124661 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.124644 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" Nov 03 18:44:46.198707 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.198674 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvrl\" (UniqueName: \"kubernetes.io/projected/7995752d-b5e3-4b41-b7e7-e9893ca31b33-kube-api-access-thvrl\") pod \"authorino-operator-7b7bb4bc77-pj4wz\" (UID: \"7995752d-b5e3-4b41-b7e7-e9893ca31b33\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" Nov 03 18:44:46.232714 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.232673 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-b5452\"" Nov 03 18:44:46.232714 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.232673 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 18:44:46.232906 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.232684 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 18:44:46.234824 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.234805 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz"] Nov 03 18:44:46.299196 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.299167 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvrl\" (UniqueName: \"kubernetes.io/projected/7995752d-b5e3-4b41-b7e7-e9893ca31b33-kube-api-access-thvrl\") pod \"authorino-operator-7b7bb4bc77-pj4wz\" (UID: \"7995752d-b5e3-4b41-b7e7-e9893ca31b33\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" Nov 03 18:44:46.363898 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.363860 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvrl\" (UniqueName: \"kubernetes.io/projected/7995752d-b5e3-4b41-b7e7-e9893ca31b33-kube-api-access-thvrl\") pod \"authorino-operator-7b7bb4bc77-pj4wz\" (UID: \"7995752d-b5e3-4b41-b7e7-e9893ca31b33\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" Nov 03 18:44:46.435833 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.435805 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" Nov 03 18:44:46.823655 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:46.823632 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz"] Nov 03 18:44:46.825275 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:44:46.825248 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7995752d_b5e3_4b41_b7e7_e9893ca31b33.slice/crio-80806372a2d745a9029baa1a83c20c006a2c8b62aedae7c2a9fd913a2050518a WatchSource:0}: Error finding container 80806372a2d745a9029baa1a83c20c006a2c8b62aedae7c2a9fd913a2050518a: Status 404 returned error can't find the container with id 80806372a2d745a9029baa1a83c20c006a2c8b62aedae7c2a9fd913a2050518a Nov 03 18:44:47.124603 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:47.124516 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" event={"ID":"7995752d-b5e3-4b41-b7e7-e9893ca31b33","Type":"ContainerStarted","Data":"80806372a2d745a9029baa1a83c20c006a2c8b62aedae7c2a9fd913a2050518a"} Nov 03 18:44:49.088667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:49.088635 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" Nov 03 18:44:50.153815 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.153780 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb"] Nov 03 18:44:50.163386 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.163353 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" Nov 03 18:44:50.214445 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.214413 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb"] Nov 03 18:44:50.217159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.217138 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tctkv\"" Nov 03 18:44:50.232510 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.232476 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnlz6\" (UniqueName: \"kubernetes.io/projected/be9f4c2f-1332-4a3f-a756-96959733fa78-kube-api-access-mnlz6\") pod \"limitador-operator-controller-manager-5778bb4449-nmztb\" (UID: \"be9f4c2f-1332-4a3f-a756-96959733fa78\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" Nov 03 18:44:50.333123 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.333007 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnlz6\" (UniqueName: \"kubernetes.io/projected/be9f4c2f-1332-4a3f-a756-96959733fa78-kube-api-access-mnlz6\") pod \"limitador-operator-controller-manager-5778bb4449-nmztb\" (UID: \"be9f4c2f-1332-4a3f-a756-96959733fa78\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" Nov 03 18:44:50.396146 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.396114 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnlz6\" (UniqueName: \"kubernetes.io/projected/be9f4c2f-1332-4a3f-a756-96959733fa78-kube-api-access-mnlz6\") pod \"limitador-operator-controller-manager-5778bb4449-nmztb\" (UID: \"be9f4c2f-1332-4a3f-a756-96959733fa78\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" Nov 03 18:44:50.480947 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.480914 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" Nov 03 18:44:50.759118 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.759051 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms"] Nov 03 18:44:50.762164 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.762145 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:50.816765 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.816732 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:50.816765 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.816738 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:50.816955 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.816771 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fnwpr\"" Nov 03 18:44:50.836434 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.836402 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168c8977-e297-4c10-b641-345254123876-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-gfbms\" (UID: \"168c8977-e297-4c10-b641-345254123876\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:50.836576 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.836458 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88wvg\" (UniqueName: \"kubernetes.io/projected/168c8977-e297-4c10-b641-345254123876-kube-api-access-88wvg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-gfbms\" (UID: \"168c8977-e297-4c10-b641-345254123876\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:50.864980 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.864953 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms"] Nov 03 18:44:50.930042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.930014 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb"] Nov 03 18:44:50.931677 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:44:50.931653 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9f4c2f_1332_4a3f_a756_96959733fa78.slice/crio-cc6ffaf1948a7a896b44fb2314c40d1cc9f867cf6457532387799f004cbefaed WatchSource:0}: Error finding container cc6ffaf1948a7a896b44fb2314c40d1cc9f867cf6457532387799f004cbefaed: Status 404 returned error can't find the container with id cc6ffaf1948a7a896b44fb2314c40d1cc9f867cf6457532387799f004cbefaed Nov 03 18:44:50.936921 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.936878 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wvg\" (UniqueName: \"kubernetes.io/projected/168c8977-e297-4c10-b641-345254123876-kube-api-access-88wvg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-gfbms\" (UID: \"168c8977-e297-4c10-b641-345254123876\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:50.937034 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.936969 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168c8977-e297-4c10-b641-345254123876-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-gfbms\" (UID: \"168c8977-e297-4c10-b641-345254123876\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:50.937320 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:50.937299 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168c8977-e297-4c10-b641-345254123876-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-gfbms\" (UID: \"168c8977-e297-4c10-b641-345254123876\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:51.002068 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:51.002037 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88wvg\" (UniqueName: \"kubernetes.io/projected/168c8977-e297-4c10-b641-345254123876-kube-api-access-88wvg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-gfbms\" (UID: \"168c8977-e297-4c10-b641-345254123876\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:51.073050 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:51.072971 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" Nov 03 18:44:51.146430 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:51.146396 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" event={"ID":"7995752d-b5e3-4b41-b7e7-e9893ca31b33","Type":"ContainerStarted","Data":"22554e9d565b12e7c2f22d5d3c519be6de6de69d236970513abe9e6687bf5a75"} Nov 03 18:44:51.146629 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:51.146608 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" Nov 03 18:44:51.147933 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:51.147912 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" event={"ID":"be9f4c2f-1332-4a3f-a756-96959733fa78","Type":"ContainerStarted","Data":"cc6ffaf1948a7a896b44fb2314c40d1cc9f867cf6457532387799f004cbefaed"} Nov 03 18:44:51.266123 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:51.266067 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" podStartSLOduration=1.578698731 podStartE2EDuration="5.266052468s" totalImagesPullingTime="3.687353737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:51.264088354 +0000 UTC m=+420.747084272" watchObservedRunningTime="2026-11-03 18:44:51.266052468 +0000 UTC m=+420.749048382" Nov 03 18:44:51.509349 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:51.509324 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms"] Nov 03 18:44:51.511207 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:44:51.511176 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168c8977_e297_4c10_b641_345254123876.slice/crio-29642a144e77f46cb54067088638e3f41e35b37bc68bd32286882dd69a79b67a WatchSource:0}: Error finding container 29642a144e77f46cb54067088638e3f41e35b37bc68bd32286882dd69a79b67a: Status 404 returned error can't find the container with id 29642a144e77f46cb54067088638e3f41e35b37bc68bd32286882dd69a79b67a Nov 03 18:44:52.154874 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:52.154834 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" event={"ID":"168c8977-e297-4c10-b641-345254123876","Type":"ContainerStarted","Data":"29642a144e77f46cb54067088638e3f41e35b37bc68bd32286882dd69a79b67a"} Nov 03 18:44:55.172959 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:55.172918 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" event={"ID":"168c8977-e297-4c10-b641-345254123876","Type":"ContainerStarted","Data":"2bd4fb7f022803b12702b703a60e3c2b42988fc014cbec1cebb04c0cef0a525b"} Nov 03 18:44:55.174575 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:55.174543 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" event={"ID":"be9f4c2f-1332-4a3f-a756-96959733fa78","Type":"ContainerStarted","Data":"1f1575b48ff8935ea7fc98f163d088b57cd1ad36265cc489dcfbfc20d3e0e1af"} Nov 03 18:44:55.174715 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:55.174677 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" Nov 03 18:44:55.416595 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:55.416534 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-gfbms" podStartSLOduration=2.519046398 podStartE2EDuration="5.416519314s" totalImagesPullingTime="2.897472916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:55.365418344 +0000 UTC m=+424.848414257" watchObservedRunningTime="2026-11-03 18:44:55.416519314 +0000 UTC m=+424.899515229" Nov 03 18:44:55.490294 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:55.490246 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" podStartSLOduration=2.066053935 podStartE2EDuration="5.490232359s" totalImagesPullingTime="3.424178424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:55.489125443 +0000 UTC m=+424.972121357" watchObservedRunningTime="2026-11-03 18:44:55.490232359 +0000 UTC m=+424.973228273" Nov 03 18:44:58.005076 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.005038 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr"] Nov 03 18:44:58.008064 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.008046 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.043271 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.043241 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr"] Nov 03 18:44:58.092454 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.092423 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mfqs9\"" Nov 03 18:44:58.096758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.096733 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkd2j\" (UniqueName: \"kubernetes.io/projected/ab6a451a-e52a-4951-a1b7-02fa91d55d02-kube-api-access-vkd2j\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brr\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.096859 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.096788 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab6a451a-e52a-4951-a1b7-02fa91d55d02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brr\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.198169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.198133 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkd2j\" (UniqueName: \"kubernetes.io/projected/ab6a451a-e52a-4951-a1b7-02fa91d55d02-kube-api-access-vkd2j\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brr\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.198345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.198200 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab6a451a-e52a-4951-a1b7-02fa91d55d02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brr\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.198609 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.198587 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab6a451a-e52a-4951-a1b7-02fa91d55d02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brr\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.264408 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.264344 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkd2j\" (UniqueName: \"kubernetes.io/projected/ab6a451a-e52a-4951-a1b7-02fa91d55d02-kube-api-access-vkd2j\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brr\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.319326 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.319290 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:44:58.752238 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:44:58.752203 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6a451a_e52a_4951_a1b7_02fa91d55d02.slice/crio-9c205ae2dad185411fce49793d79baa3e26acf2b7920c59e7490ca1db03d5602 WatchSource:0}: Error finding container 9c205ae2dad185411fce49793d79baa3e26acf2b7920c59e7490ca1db03d5602: Status 404 returned error can't find the container with id 9c205ae2dad185411fce49793d79baa3e26acf2b7920c59e7490ca1db03d5602 Nov 03 18:44:58.752371 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:58.752300 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr"] Nov 03 18:44:59.195085 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:44:59.195053 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" event={"ID":"ab6a451a-e52a-4951-a1b7-02fa91d55d02","Type":"ContainerStarted","Data":"9c205ae2dad185411fce49793d79baa3e26acf2b7920c59e7490ca1db03d5602"} Nov 03 18:45:01.646654 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.646614 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j"] Nov 03 18:45:01.649891 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.649873 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" Nov 03 18:45:01.703885 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.703852 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8mj5j\"" Nov 03 18:45:01.728463 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.728403 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkv72\" (UniqueName: \"kubernetes.io/projected/3bb8de85-fed2-48d4-aeda-c1e45945aeeb-kube-api-access-qkv72\") pod \"dns-operator-controller-manager-8557cbdb84-rfj7j\" (UID: \"3bb8de85-fed2-48d4-aeda-c1e45945aeeb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" Nov 03 18:45:01.754402 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.754368 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 18:45:01.757676 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.757642 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j"] Nov 03 18:45:01.829831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.829789 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkv72\" (UniqueName: \"kubernetes.io/projected/3bb8de85-fed2-48d4-aeda-c1e45945aeeb-kube-api-access-qkv72\") pod \"dns-operator-controller-manager-8557cbdb84-rfj7j\" (UID: \"3bb8de85-fed2-48d4-aeda-c1e45945aeeb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" Nov 03 18:45:01.897691 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.897621 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkv72\" (UniqueName: \"kubernetes.io/projected/3bb8de85-fed2-48d4-aeda-c1e45945aeeb-kube-api-access-qkv72\") pod \"dns-operator-controller-manager-8557cbdb84-rfj7j\" (UID: \"3bb8de85-fed2-48d4-aeda-c1e45945aeeb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" Nov 03 18:45:01.962388 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:01.962349 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" Nov 03 18:45:02.157317 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:02.157229 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pj4wz" Nov 03 18:45:02.397431 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:02.397407 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j"] Nov 03 18:45:02.399561 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:45:02.399538 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb8de85_fed2_48d4_aeda_c1e45945aeeb.slice/crio-763d56c0069c6df130ab2782d37749ea50958756063d07552db6411cdb6ab46a WatchSource:0}: Error finding container 763d56c0069c6df130ab2782d37749ea50958756063d07552db6411cdb6ab46a: Status 404 returned error can't find the container with id 763d56c0069c6df130ab2782d37749ea50958756063d07552db6411cdb6ab46a Nov 03 18:45:02.792165 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:45:02.792106 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:45:03.218992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:03.218955 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" event={"ID":"3bb8de85-fed2-48d4-aeda-c1e45945aeeb","Type":"ContainerStarted","Data":"763d56c0069c6df130ab2782d37749ea50958756063d07552db6411cdb6ab46a"} Nov 03 18:45:04.920131 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:04.917352 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp"] Nov 03 18:45:04.924078 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:04.924050 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.001663 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.001624 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp"] Nov 03 18:45:05.051710 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.051662 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5vztq\"" Nov 03 18:45:05.066053 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066024 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066189 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066059 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066189 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066078 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066189 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066154 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjhh\" (UniqueName: \"kubernetes.io/projected/38f23fde-7287-45c5-9e3a-31a0ae356e1d-kube-api-access-mqjhh\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066294 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066220 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066294 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066243 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066354 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066304 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066354 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066330 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.066354 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.066351 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167499 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167469 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjhh\" (UniqueName: \"kubernetes.io/projected/38f23fde-7287-45c5-9e3a-31a0ae356e1d-kube-api-access-mqjhh\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167642 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167517 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167642 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167624 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167680 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167747 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167821 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167777 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167869 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167853 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167904 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167889 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.167938 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.167918 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.168125 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.168101 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.168199 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.168164 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.168289 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.168267 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.168369 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.168343 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.168415 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.168385 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.169964 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.169937 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.170052 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.170013 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.230375 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.230343 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f23fde-7287-45c5-9e3a-31a0ae356e1d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.230633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.230615 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjhh\" (UniqueName: \"kubernetes.io/projected/38f23fde-7287-45c5-9e3a-31a0ae356e1d-kube-api-access-mqjhh\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp\" (UID: \"38f23fde-7287-45c5-9e3a-31a0ae356e1d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.237649 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.237449 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:05.739672 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.739643 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp"] Nov 03 18:45:05.741995 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:45:05.741960 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f23fde_7287_45c5_9e3a_31a0ae356e1d.slice/crio-b3f693f5d7b3b62f67dc121101a0b5765bb3273a3d1912820ade3e3d088c65bc WatchSource:0}: Error finding container b3f693f5d7b3b62f67dc121101a0b5765bb3273a3d1912820ade3e3d088c65bc: Status 404 returned error can't find the container with id b3f693f5d7b3b62f67dc121101a0b5765bb3273a3d1912820ade3e3d088c65bc Nov 03 18:45:05.744287 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.744251 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:45:05.744380 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.744333 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:45:05.744431 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:05.744380 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:45:06.182186 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:06.182114 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nmztb" Nov 03 18:45:06.237950 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:06.237918 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" event={"ID":"38f23fde-7287-45c5-9e3a-31a0ae356e1d","Type":"ContainerStarted","Data":"5772442db3de7f675ab665a61832c04a5a621327e71764bd3416350e360a5447"} Nov 03 18:45:06.238096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:06.237965 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" event={"ID":"38f23fde-7287-45c5-9e3a-31a0ae356e1d","Type":"ContainerStarted","Data":"b3f693f5d7b3b62f67dc121101a0b5765bb3273a3d1912820ade3e3d088c65bc"} Nov 03 18:45:06.424802 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:06.424754 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" podStartSLOduration=2.424739338 podStartE2EDuration="2.424739338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:06.422838244 +0000 UTC m=+435.905834158" watchObservedRunningTime="2026-11-03 18:45:06.424739338 +0000 UTC m=+435.907735251" Nov 03 18:45:07.238763 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:07.238728 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:07.246488 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:07.246461 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:08.250714 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:08.250668 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:08.251653 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:08.251635 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp" Nov 03 18:45:09.257162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:09.257126 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" event={"ID":"ab6a451a-e52a-4951-a1b7-02fa91d55d02","Type":"ContainerStarted","Data":"e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5"} Nov 03 18:45:09.486769 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:09.486650 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" podStartSLOduration=2.75016496 podStartE2EDuration="12.486635173s" totalImagesPullingTime="9.736470213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:09.48535685 +0000 UTC m=+438.968352765" watchObservedRunningTime="2026-11-03 18:45:09.486635173 +0000 UTC m=+438.969631086" Nov 03 18:45:10.263890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:10.263845 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" event={"ID":"3bb8de85-fed2-48d4-aeda-c1e45945aeeb","Type":"ContainerStarted","Data":"3d762bc42c3e7c4fc8385a3fea3eab7c1a89d7670c4d96258c759bbbfe809ba5"} Nov 03 18:45:10.264340 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:10.263992 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:45:10.395172 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:10.395117 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" podStartSLOduration=2.5953285619999997 podStartE2EDuration="9.395104463s" totalImagesPullingTime="6.799775901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:10.393114123 +0000 UTC m=+439.876110037" watchObservedRunningTime="2026-11-03 18:45:10.395104463 +0000 UTC m=+439.878100377" Nov 03 18:45:11.268668 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:11.268627 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" Nov 03 18:45:12.973284 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:12.973251 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9pbs9"] Nov 03 18:45:12.975863 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:12.975846 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" Nov 03 18:45:13.029472 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:13.029442 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9pbs9"] Nov 03 18:45:13.078187 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:13.078162 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2q9wr\"" Nov 03 18:45:13.136963 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:13.136932 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtvp\" (UniqueName: \"kubernetes.io/projected/125f5631-9e23-423a-8300-3a6248b2ef75-kube-api-access-gwtvp\") pod \"authorino-57f7f8659f-9pbs9\" (UID: \"125f5631-9e23-423a-8300-3a6248b2ef75\") " pod="kuadrant-system/authorino-57f7f8659f-9pbs9" Nov 03 18:45:13.237872 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:13.237781 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtvp\" (UniqueName: \"kubernetes.io/projected/125f5631-9e23-423a-8300-3a6248b2ef75-kube-api-access-gwtvp\") pod \"authorino-57f7f8659f-9pbs9\" (UID: \"125f5631-9e23-423a-8300-3a6248b2ef75\") " pod="kuadrant-system/authorino-57f7f8659f-9pbs9" Nov 03 18:45:13.299968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:13.299943 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtvp\" (UniqueName: \"kubernetes.io/projected/125f5631-9e23-423a-8300-3a6248b2ef75-kube-api-access-gwtvp\") pod \"authorino-57f7f8659f-9pbs9\" (UID: \"125f5631-9e23-423a-8300-3a6248b2ef75\") " pod="kuadrant-system/authorino-57f7f8659f-9pbs9" Nov 03 18:45:13.597439 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:13.597361 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" Nov 03 18:45:13.972550 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:13.972524 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9pbs9"] Nov 03 18:45:13.974023 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:45:13.973997 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod125f5631_9e23_423a_8300_3a6248b2ef75.slice/crio-1956e51f239ca9026ce99a270467cd37fe97bbe4dd33d6a45d9ba1d5e09459b1 WatchSource:0}: Error finding container 1956e51f239ca9026ce99a270467cd37fe97bbe4dd33d6a45d9ba1d5e09459b1: Status 404 returned error can't find the container with id 1956e51f239ca9026ce99a270467cd37fe97bbe4dd33d6a45d9ba1d5e09459b1 Nov 03 18:45:14.284506 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:14.284424 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" event={"ID":"125f5631-9e23-423a-8300-3a6248b2ef75","Type":"ContainerStarted","Data":"1956e51f239ca9026ce99a270467cd37fe97bbe4dd33d6a45d9ba1d5e09459b1"} Nov 03 18:45:15.922083 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:15.922048 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l6rzf"] Nov 03 18:45:15.924581 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:15.924558 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" Nov 03 18:45:15.988064 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:15.988028 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l6rzf"] Nov 03 18:45:16.060774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:16.060737 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mmh\" (UniqueName: \"kubernetes.io/projected/73d34ee2-ccdb-4008-af49-563f2a910064-kube-api-access-55mmh\") pod \"authorino-56c6c9ddd5-l6rzf\" (UID: \"73d34ee2-ccdb-4008-af49-563f2a910064\") " pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" Nov 03 18:45:16.163084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:16.162430 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mmh\" (UniqueName: \"kubernetes.io/projected/73d34ee2-ccdb-4008-af49-563f2a910064-kube-api-access-55mmh\") pod \"authorino-56c6c9ddd5-l6rzf\" (UID: \"73d34ee2-ccdb-4008-af49-563f2a910064\") " pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" Nov 03 18:45:16.225018 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:16.224984 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mmh\" (UniqueName: \"kubernetes.io/projected/73d34ee2-ccdb-4008-af49-563f2a910064-kube-api-access-55mmh\") pod \"authorino-56c6c9ddd5-l6rzf\" (UID: \"73d34ee2-ccdb-4008-af49-563f2a910064\") " pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" Nov 03 18:45:16.239731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:16.239689 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" Nov 03 18:45:16.622197 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:16.622126 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l6rzf"] Nov 03 18:45:16.625319 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:45:16.625273 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73d34ee2_ccdb_4008_af49_563f2a910064.slice/crio-c897f60dc842fd69e689c52ce2fa3a68cc4b2a4fd8cd01079e12b36dbe7906d8 WatchSource:0}: Error finding container c897f60dc842fd69e689c52ce2fa3a68cc4b2a4fd8cd01079e12b36dbe7906d8: Status 404 returned error can't find the container with id c897f60dc842fd69e689c52ce2fa3a68cc4b2a4fd8cd01079e12b36dbe7906d8 Nov 03 18:45:17.302003 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:17.301968 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" event={"ID":"73d34ee2-ccdb-4008-af49-563f2a910064","Type":"ContainerStarted","Data":"c897f60dc842fd69e689c52ce2fa3a68cc4b2a4fd8cd01079e12b36dbe7906d8"} Nov 03 18:45:17.303469 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:17.303442 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" event={"ID":"125f5631-9e23-423a-8300-3a6248b2ef75","Type":"ContainerStarted","Data":"7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc"} Nov 03 18:45:17.420343 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:17.420289 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" podStartSLOduration=2.725078217 podStartE2EDuration="5.420275795s" totalImagesPullingTime="2.695197578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:17.418817807 +0000 UTC m=+446.901813722" watchObservedRunningTime="2026-11-03 18:45:17.420275795 +0000 UTC m=+446.903271709" Nov 03 18:45:18.310015 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:18.309982 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" event={"ID":"73d34ee2-ccdb-4008-af49-563f2a910064","Type":"ContainerStarted","Data":"34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f"} Nov 03 18:45:18.427303 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:18.427241 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" podStartSLOduration=2.160586055 podStartE2EDuration="3.427227132s" totalImagesPullingTime="1.266641077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:18.425108004 +0000 UTC m=+447.908103918" watchObservedRunningTime="2026-11-03 18:45:18.427227132 +0000 UTC m=+447.910223064" Nov 03 18:45:18.478100 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:18.478065 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9pbs9"] Nov 03 18:45:19.314827 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:19.314789 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" podUID="125f5631-9e23-423a-8300-3a6248b2ef75" containerName="authorino" containerID="cri-o://7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc" gracePeriod=30 Nov 03 18:45:19.559591 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:19.559564 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" Nov 03 18:45:19.691964 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:19.691935 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125f5631-9e23-423a-8300-3a6248b2ef75-kube-api-access-gwtvp\" (UniqueName: \"kubernetes.io/projected/125f5631-9e23-423a-8300-3a6248b2ef75-kube-api-access-gwtvp\") pod \"125f5631-9e23-423a-8300-3a6248b2ef75\" (UID: \"125f5631-9e23-423a-8300-3a6248b2ef75\") " Nov 03 18:45:19.694217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:19.694179 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125f5631-9e23-423a-8300-3a6248b2ef75-kube-api-access-gwtvp" pod "125f5631-9e23-423a-8300-3a6248b2ef75" (UID: "125f5631-9e23-423a-8300-3a6248b2ef75"). InnerVolumeSpecName "kube-api-access-gwtvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:19.793407 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:19.793375 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwtvp\" (UniqueName: \"kubernetes.io/projected/125f5631-9e23-423a-8300-3a6248b2ef75-kube-api-access-gwtvp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:20.320992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.320953 4948 generic.go:356] "Generic (PLEG): container finished" podID="125f5631-9e23-423a-8300-3a6248b2ef75" containerID="7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc" exitCode=0 Nov 03 18:45:20.320992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.320994 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" Nov 03 18:45:20.321521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.321042 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" event={"ID":"125f5631-9e23-423a-8300-3a6248b2ef75","Type":"ContainerDied","Data":"7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc"} Nov 03 18:45:20.321521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.321096 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9pbs9" event={"ID":"125f5631-9e23-423a-8300-3a6248b2ef75","Type":"ContainerDied","Data":"1956e51f239ca9026ce99a270467cd37fe97bbe4dd33d6a45d9ba1d5e09459b1"} Nov 03 18:45:20.321521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.321121 4948 scope.go:122] "RemoveContainer" containerID="7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc" Nov 03 18:45:20.330614 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.330597 4948 scope.go:122] "RemoveContainer" containerID="7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc" Nov 03 18:45:20.330902 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:45:20.330881 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc\": container with ID starting with 7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc not found: ID does not exist" containerID="7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc" Nov 03 18:45:20.330955 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.330912 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc"} err="failed to get container status \"7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc\": rpc error: code = NotFound desc = could not find container \"7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc\": container with ID starting with 7c5ccd9373aef027f502e12980366054788d753effe4eeac772a4f41ce6455bc not found: ID does not exist" Nov 03 18:45:20.488761 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.488728 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9pbs9"] Nov 03 18:45:20.492229 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.492207 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9pbs9"] Nov 03 18:45:20.752043 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:20.752011 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125f5631-9e23-423a-8300-3a6248b2ef75" path="/var/lib/kubelet/pods/125f5631-9e23-423a-8300-3a6248b2ef75/volumes" Nov 03 18:45:21.271250 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:21.271223 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 18:45:22.275868 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:22.275840 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rfj7j" Nov 03 18:45:31.233091 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.233054 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr"] Nov 03 18:45:31.233481 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.233467 4948 cpu_manager.go:414] "containerMap: removing container" podUID="125f5631-9e23-423a-8300-3a6248b2ef75" containerName="authorino" Nov 03 18:45:31.233525 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.233482 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125f5631-9e23-423a-8300-3a6248b2ef75" containerName="authorino" Nov 03 18:45:31.233574 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.233564 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="125f5631-9e23-423a-8300-3a6248b2ef75" containerName="authorino" Nov 03 18:45:31.238358 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.238339 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.287975 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.287942 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82be7705-1844-40b4-9349-49951cf7962e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.287975 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.287973 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82be7705-1844-40b4-9349-49951cf7962e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.288162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.288069 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82be7705-1844-40b4-9349-49951cf7962e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.288162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.288148 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2v85\" (UniqueName: \"kubernetes.io/projected/82be7705-1844-40b4-9349-49951cf7962e-kube-api-access-c2v85\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.342845 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.342811 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 18:45:31.342845 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.342811 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-hbzm4\"" Nov 03 18:45:31.342845 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.342820 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 18:45:31.388824 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.388793 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82be7705-1844-40b4-9349-49951cf7962e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.388973 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.388839 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2v85\" (UniqueName: \"kubernetes.io/projected/82be7705-1844-40b4-9349-49951cf7962e-kube-api-access-c2v85\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.388973 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.388876 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82be7705-1844-40b4-9349-49951cf7962e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.388973 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.388894 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82be7705-1844-40b4-9349-49951cf7962e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.389257 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.389234 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82be7705-1844-40b4-9349-49951cf7962e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.389300 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.389258 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82be7705-1844-40b4-9349-49951cf7962e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.445906 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.445874 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 18:45:31.448166 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.448146 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr"] Nov 03 18:45:31.450633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.450608 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82be7705-1844-40b4-9349-49951cf7962e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.452765 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.452744 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2v85\" (UniqueName: \"kubernetes.io/projected/82be7705-1844-40b4-9349-49951cf7962e-kube-api-access-c2v85\") pod \"kserve-module-controller-manager-7566cf5fc8-9rqtr\" (UID: \"82be7705-1844-40b4-9349-49951cf7962e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.550008 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.549924 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:31.986980 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:31.986958 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr"] Nov 03 18:45:31.989036 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:45:31.989003 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82be7705_1844_40b4_9349_49951cf7962e.slice/crio-eb0ea72e3c0391d9db726f047e8fce62299e8555127c268c52ad76f735c98231 WatchSource:0}: Error finding container eb0ea72e3c0391d9db726f047e8fce62299e8555127c268c52ad76f735c98231: Status 404 returned error can't find the container with id eb0ea72e3c0391d9db726f047e8fce62299e8555127c268c52ad76f735c98231 Nov 03 18:45:32.385216 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:32.385125 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" event={"ID":"82be7705-1844-40b4-9349-49951cf7962e","Type":"ContainerStarted","Data":"eb0ea72e3c0391d9db726f047e8fce62299e8555127c268c52ad76f735c98231"} Nov 03 18:45:32.792527 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:45:32.792489 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:45:35.406255 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:35.406215 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" event={"ID":"82be7705-1844-40b4-9349-49951cf7962e","Type":"ContainerStarted","Data":"b9949cfbf4ccd9124fc9b971467ef797eda73a563dc4f7b2f7ca884846f69541"} Nov 03 18:45:35.406786 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:35.406334 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:35.580248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:35.580196 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" podStartSLOduration=1.93529586 podStartE2EDuration="4.580182009s" totalImagesPullingTime="2.644886149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:35.58001445 +0000 UTC m=+465.063010363" watchObservedRunningTime="2026-11-03 18:45:35.580182009 +0000 UTC m=+465.063177938" Nov 03 18:45:56.413093 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:56.413063 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9rqtr" Nov 03 18:45:57.199016 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.198982 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ftfkh"] Nov 03 18:45:57.205478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.205456 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.259027 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.258991 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 18:45:57.259240 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.259219 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mzt96\"" Nov 03 18:45:57.259319 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.259291 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 18:45:57.259642 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.259624 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 18:45:57.307233 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.307198 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-metrics-certs\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.307383 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.307261 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhlc\" (UniqueName: \"kubernetes.io/projected/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-kube-api-access-2qhlc\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.307424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.307374 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-webhook-cert\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.312519 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.312499 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ftfkh"] Nov 03 18:45:57.407881 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.407849 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-metrics-certs\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.408069 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.407908 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhlc\" (UniqueName: \"kubernetes.io/projected/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-kube-api-access-2qhlc\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.408069 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.407945 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-webhook-cert\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.411019 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.410995 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-webhook-cert\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.411019 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.411025 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-metrics-certs\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.470091 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.470021 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhlc\" (UniqueName: \"kubernetes.io/projected/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-kube-api-access-2qhlc\") pod \"maas-controller-579cf7c6d-ftfkh\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.519795 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.519752 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:45:57.632137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.632108 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68f76587d6-m4k75"] Nov 03 18:45:57.637744 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.637723 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.692969 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.692938 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68f76587d6-m4k75"] Nov 03 18:45:57.710348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.710320 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgf9h\" (UniqueName: \"kubernetes.io/projected/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-kube-api-access-xgf9h\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.710501 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.710480 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-webhook-cert\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.710560 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.710510 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-metrics-certs\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.811665 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.811543 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-webhook-cert\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.811665 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.811655 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-metrics-certs\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.811898 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.811727 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgf9h\" (UniqueName: \"kubernetes.io/projected/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-kube-api-access-xgf9h\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.819261 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.819228 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-metrics-certs\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.819386 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.819337 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-webhook-cert\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.873969 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.873940 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgf9h\" (UniqueName: \"kubernetes.io/projected/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-kube-api-access-xgf9h\") pod \"maas-controller-68f76587d6-m4k75\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:57.893801 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.893778 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ftfkh"] Nov 03 18:45:57.895303 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:45:57.895280 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c98eb44_7fc2_41a5_9e6d_0132e7d9cac8.slice/crio-d1478ef044e0f244cd8a204ccea320c727633be74e2ee5d6224319e2502cad1e WatchSource:0}: Error finding container d1478ef044e0f244cd8a204ccea320c727633be74e2ee5d6224319e2502cad1e: Status 404 returned error can't find the container with id d1478ef044e0f244cd8a204ccea320c727633be74e2ee5d6224319e2502cad1e Nov 03 18:45:57.951067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:57.951035 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:45:58.326281 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:58.326257 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68f76587d6-m4k75"] Nov 03 18:45:58.328053 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:45:58.328025 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aaae9b4_e7dc_4cdb_bb80_9a4639354338.slice/crio-815bfeb0a5ec7ba5785682571e561b441945ce8e585230ac1b1bb91bfb342bb3 WatchSource:0}: Error finding container 815bfeb0a5ec7ba5785682571e561b441945ce8e585230ac1b1bb91bfb342bb3: Status 404 returned error can't find the container with id 815bfeb0a5ec7ba5785682571e561b441945ce8e585230ac1b1bb91bfb342bb3 Nov 03 18:45:58.532992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:58.532955 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-m4k75" event={"ID":"9aaae9b4-e7dc-4cdb-bb80-9a4639354338","Type":"ContainerStarted","Data":"815bfeb0a5ec7ba5785682571e561b441945ce8e585230ac1b1bb91bfb342bb3"} Nov 03 18:45:58.534285 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:45:58.534248 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" event={"ID":"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8","Type":"ContainerStarted","Data":"d1478ef044e0f244cd8a204ccea320c727633be74e2ee5d6224319e2502cad1e"} Nov 03 18:46:02.563384 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:02.563344 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-m4k75" event={"ID":"9aaae9b4-e7dc-4cdb-bb80-9a4639354338","Type":"ContainerStarted","Data":"1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7"} Nov 03 18:46:02.563860 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:02.563436 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:46:02.565010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:02.564980 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" event={"ID":"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8","Type":"ContainerStarted","Data":"05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8"} Nov 03 18:46:02.565146 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:02.565078 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:46:02.694131 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:02.694080 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68f76587d6-m4k75" podStartSLOduration=2.333633422 podStartE2EDuration="5.694066337s" totalImagesPullingTime="3.360432915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:02.693357524 +0000 UTC m=+492.176353438" watchObservedRunningTime="2026-11-03 18:46:02.694066337 +0000 UTC m=+492.177062251" Nov 03 18:46:02.792192 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:46:02.792156 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:46:02.831152 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:02.831048 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" podStartSLOduration=2.176310177 podStartE2EDuration="5.831034782s" totalImagesPullingTime="3.654724605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:02.82843012 +0000 UTC m=+492.311426034" watchObservedRunningTime="2026-11-03 18:46:02.831034782 +0000 UTC m=+492.314030695" Nov 03 18:46:13.572275 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:13.572242 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:46:13.572818 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:13.572324 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:46:13.746016 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:13.745980 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ftfkh"] Nov 03 18:46:13.746192 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:13.746167 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" podUID="5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" containerName="manager" containerID="cri-o://05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8" gracePeriod=30 Nov 03 18:46:13.991074 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:13.991051 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:46:14.161831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.161689 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-metrics-certs\") pod \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " Nov 03 18:46:14.161998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.161886 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-kube-api-access-2qhlc\" (UniqueName: \"kubernetes.io/projected/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-kube-api-access-2qhlc\") pod \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " Nov 03 18:46:14.161998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.161913 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-webhook-cert\") pod \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\" (UID: \"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8\") " Nov 03 18:46:14.164116 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.164082 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-metrics-certs" pod "5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" (UID: "5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:46:14.164233 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.164144 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-kube-api-access-2qhlc" pod "5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" (UID: "5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8"). InnerVolumeSpecName "kube-api-access-2qhlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:46:14.164311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.164292 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-webhook-cert" pod "5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" (UID: "5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:46:14.263579 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.263539 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qhlc\" (UniqueName: \"kubernetes.io/projected/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-kube-api-access-2qhlc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:14.263579 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.263574 4948 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-webhook-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:14.263579 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.263584 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:14.630103 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.630071 4948 generic.go:356] "Generic (PLEG): container finished" podID="5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" containerID="05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8" exitCode=0 Nov 03 18:46:14.630583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.630121 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" Nov 03 18:46:14.630583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.630161 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" event={"ID":"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8","Type":"ContainerDied","Data":"05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8"} Nov 03 18:46:14.630583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.630223 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ftfkh" event={"ID":"5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8","Type":"ContainerDied","Data":"d1478ef044e0f244cd8a204ccea320c727633be74e2ee5d6224319e2502cad1e"} Nov 03 18:46:14.630583 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.630257 4948 scope.go:122] "RemoveContainer" containerID="05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8" Nov 03 18:46:14.640201 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.640184 4948 scope.go:122] "RemoveContainer" containerID="05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8" Nov 03 18:46:14.640456 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:46:14.640438 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8\": container with ID starting with 05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8 not found: ID does not exist" containerID="05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8" Nov 03 18:46:14.640522 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.640466 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8"} err="failed to get container status \"05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8\": rpc error: code = NotFound desc = could not find container \"05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8\": container with ID starting with 05e0f09f152de9ca5acc3bce71f422740e6409c59b50cc02d08a78ef1a8d68d8 not found: ID does not exist" Nov 03 18:46:14.798820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.798787 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ftfkh"] Nov 03 18:46:14.802498 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:14.802475 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ftfkh"] Nov 03 18:46:16.751643 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:16.751608 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" path="/var/lib/kubelet/pods/5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8/volumes" Nov 03 18:46:32.793239 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:46:32.793201 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:46:32.937439 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:32.937407 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk"] Nov 03 18:46:32.937800 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:32.937787 4948 cpu_manager.go:414] "containerMap: removing container" podUID="5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" containerName="manager" Nov 03 18:46:32.937855 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:32.937801 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" containerName="manager" Nov 03 18:46:32.937889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:32.937868 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c98eb44-7fc2-41a5-9e6d-0132e7d9cac8" containerName="manager" Nov 03 18:46:32.940614 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:32.940599 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.000103 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.000068 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk"] Nov 03 18:46:33.022992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.022961 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.022992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.022992 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.023202 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.023012 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.023202 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.023029 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7gp\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-kube-api-access-jw7gp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.023202 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.023184 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e929ed-5855-4d0e-98f3-c9c0351c53ab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.023323 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.023236 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.023323 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.023264 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.047753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.047653 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 18:46:33.047904 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.047761 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 18:46:33.047904 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.047764 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 18:46:33.047904 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.047767 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4qf6p\"" Nov 03 18:46:33.124248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.124210 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e929ed-5855-4d0e-98f3-c9c0351c53ab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.124444 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.124272 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.124444 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.124309 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.124444 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.124390 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.124444 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.124416 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.124662 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.124590 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.124662 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.124637 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7gp\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-kube-api-access-jw7gp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.125183 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.125161 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.127200 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.127176 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.127315 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.127238 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.127388 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.127323 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.127439 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.127383 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e929ed-5855-4d0e-98f3-c9c0351c53ab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.186817 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.186789 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.186994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.186962 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7gp\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-kube-api-access-jw7gp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjtdk\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.252075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.252045 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:33.683710 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.683675 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk"] Nov 03 18:46:33.684924 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:46:33.684897 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e929ed_5855_4d0e_98f3_c9c0351c53ab.slice/crio-43db91dd4f2714a1d4447bedfe202eae300c7746dcde13e3b1d867a6f2d2a555 WatchSource:0}: Error finding container 43db91dd4f2714a1d4447bedfe202eae300c7746dcde13e3b1d867a6f2d2a555: Status 404 returned error can't find the container with id 43db91dd4f2714a1d4447bedfe202eae300c7746dcde13e3b1d867a6f2d2a555 Nov 03 18:46:33.731051 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:33.731012 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" event={"ID":"47e929ed-5855-4d0e-98f3-c9c0351c53ab","Type":"ContainerStarted","Data":"43db91dd4f2714a1d4447bedfe202eae300c7746dcde13e3b1d867a6f2d2a555"} Nov 03 18:46:37.519250 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:37.519216 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:46:37.519546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:37.519287 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:46:37.759277 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:37.759247 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" event={"ID":"47e929ed-5855-4d0e-98f3-c9c0351c53ab","Type":"ContainerStarted","Data":"2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc"} Nov 03 18:46:37.759468 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:37.759361 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:37.973172 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:37.973112 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" podStartSLOduration=2.141739382 podStartE2EDuration="5.973094891s" totalImagesPullingTime="3.831355509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:37.928742489 +0000 UTC m=+527.411738406" watchObservedRunningTime="2026-11-03 18:46:37.973094891 +0000 UTC m=+527.456090806" Nov 03 18:46:38.768156 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:38.768123 4948 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qjtdk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:46:38.768544 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:38.768199 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" podUID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:46:41.767488 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:41.767452 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 18:46:52.838755 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:52.838720 4948 generic.go:356] "Generic (PLEG): container finished" podID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerID="4a22ef7fd92ba9b0a12c40408a99ebce65cc489bc9ebf24d8d43afbd65ad3c5b" exitCode=0 Nov 03 18:46:52.838755 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:52.838736 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" event={"ID":"f781701e-aa01-4167-84f2-0e2c7a7601dc","Type":"ContainerDied","Data":"4a22ef7fd92ba9b0a12c40408a99ebce65cc489bc9ebf24d8d43afbd65ad3c5b"} Nov 03 18:46:52.839213 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:52.839078 4948 scope.go:122] "RemoveContainer" containerID="4a22ef7fd92ba9b0a12c40408a99ebce65cc489bc9ebf24d8d43afbd65ad3c5b" Nov 03 18:46:52.840314 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:52.840284 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:46:52.840383 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:52.840366 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:46:52.840430 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:52.840415 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:46:53.846576 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:53.846538 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" event={"ID":"f781701e-aa01-4167-84f2-0e2c7a7601dc","Type":"ContainerStarted","Data":"c267e66b4781e6ebc33bb4eadf8bf62afd4fa6275769fe92c3eedcce731b0981"} Nov 03 18:46:54.419431 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:54.419398 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:46:54.574724 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:54.573801 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:46:54.574724 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:54.573946 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vtbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 03 18:46:54.574724 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:54.574004 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 03 18:46:55.555137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:55.555098 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vtbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 03 18:46:55.561821 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:55.561782 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 03 18:46:56.006615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.006578 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l6rzf"] Nov 03 18:46:56.006804 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.006782 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" podUID="73d34ee2-ccdb-4008-af49-563f2a910064" containerName="authorino" containerID="cri-o://34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f" gracePeriod=30 Nov 03 18:46:56.303215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.303186 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" Nov 03 18:46:56.487190 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.487150 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d34ee2-ccdb-4008-af49-563f2a910064-kube-api-access-55mmh\" (UniqueName: \"kubernetes.io/projected/73d34ee2-ccdb-4008-af49-563f2a910064-kube-api-access-55mmh\") pod \"73d34ee2-ccdb-4008-af49-563f2a910064\" (UID: \"73d34ee2-ccdb-4008-af49-563f2a910064\") " Nov 03 18:46:56.489221 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.489187 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d34ee2-ccdb-4008-af49-563f2a910064-kube-api-access-55mmh" pod "73d34ee2-ccdb-4008-af49-563f2a910064" (UID: "73d34ee2-ccdb-4008-af49-563f2a910064"). InnerVolumeSpecName "kube-api-access-55mmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:46:56.571179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.571092 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vtbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 03 18:46:56.571763 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.571734 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 03 18:46:56.588385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.588354 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55mmh\" (UniqueName: \"kubernetes.io/projected/73d34ee2-ccdb-4008-af49-563f2a910064-kube-api-access-55mmh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:46:56.891394 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.891310 4948 generic.go:356] "Generic (PLEG): container finished" podID="73d34ee2-ccdb-4008-af49-563f2a910064" containerID="34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f" exitCode=0 Nov 03 18:46:56.891394 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.891361 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" Nov 03 18:46:56.891569 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.891403 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" event={"ID":"73d34ee2-ccdb-4008-af49-563f2a910064","Type":"ContainerDied","Data":"34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f"} Nov 03 18:46:56.891569 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.891449 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l6rzf" event={"ID":"73d34ee2-ccdb-4008-af49-563f2a910064","Type":"ContainerDied","Data":"c897f60dc842fd69e689c52ce2fa3a68cc4b2a4fd8cd01079e12b36dbe7906d8"} Nov 03 18:46:56.891569 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.891474 4948 scope.go:122] "RemoveContainer" containerID="34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f" Nov 03 18:46:56.900478 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.900458 4948 scope.go:122] "RemoveContainer" containerID="34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f" Nov 03 18:46:56.900803 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:46:56.900775 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f\": container with ID starting with 34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f not found: ID does not exist" containerID="34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f" Nov 03 18:46:56.900911 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:56.900811 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f"} err="failed to get container status \"34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f\": rpc error: code = NotFound desc = could not find container \"34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f\": container with ID starting with 34cf8f7e23c7149cfc9a4d434641d7266edfd0b9505701ddd76a57a02ce63d3f not found: ID does not exist" Nov 03 18:46:57.076768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:57.068758 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l6rzf"] Nov 03 18:46:57.082678 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:57.082639 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l6rzf"] Nov 03 18:46:57.562720 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:57.557065 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vtbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 03 18:46:57.562720 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:57.557128 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 03 18:46:58.555169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:58.555125 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vtbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 03 18:46:58.555808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:58.555771 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 03 18:46:58.753536 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:58.753500 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d34ee2-ccdb-4008-af49-563f2a910064" path="/var/lib/kubelet/pods/73d34ee2-ccdb-4008-af49-563f2a910064/volumes" Nov 03 18:46:59.555975 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:59.555945 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:46:59.556972 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:46:59.556952 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 18:47:02.792472 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:47:02.792435 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:47:32.792902 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:47:32.792825 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:47:50.796123 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:47:50.796088 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:47:50.797031 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:47:50.797004 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:47:50.803025 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:47:50.803001 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:47:50.803550 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:47:50.803531 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:48:02.792537 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:48:02.792496 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:48:31.396692 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:48:31.396654 4948 generic.go:356] "Generic (PLEG): container finished" podID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerID="1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7" exitCode=1 Nov 03 18:48:31.397121 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:48:31.396729 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-m4k75" event={"ID":"9aaae9b4-e7dc-4cdb-bb80-9a4639354338","Type":"ContainerDied","Data":"1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7"} Nov 03 18:48:31.397121 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:48:31.397093 4948 scope.go:122] "RemoveContainer" containerID="1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7" Nov 03 18:48:31.397463 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:48:31.397448 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:48:32.404949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:48:32.404915 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-m4k75" event={"ID":"9aaae9b4-e7dc-4cdb-bb80-9a4639354338","Type":"ContainerStarted","Data":"1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2"} Nov 03 18:48:32.405295 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:48:32.405134 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:48:32.792833 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:48:32.792798 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:48:43.414045 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:48:43.414011 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 18:49:02.793448 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:49:02.793366 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:49:06.316136 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.316101 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg"] Nov 03 18:49:06.316747 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.316724 4948 cpu_manager.go:414] "containerMap: removing container" podUID="73d34ee2-ccdb-4008-af49-563f2a910064" containerName="authorino" Nov 03 18:49:06.316747 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.316746 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d34ee2-ccdb-4008-af49-563f2a910064" containerName="authorino" Nov 03 18:49:06.316916 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.316813 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d34ee2-ccdb-4008-af49-563f2a910064" containerName="authorino" Nov 03 18:49:06.319833 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.319817 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.404686 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.404653 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg"] Nov 03 18:49:06.409820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.409796 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c659d570-af36-4aca-9532-5dc9e24ef622-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.409919 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.409832 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lqf\" (UniqueName: \"kubernetes.io/projected/c659d570-af36-4aca-9532-5dc9e24ef622-kube-api-access-h6lqf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.409919 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.409861 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.410012 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.409989 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.410049 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.410029 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.410092 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.410052 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.431447 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.431421 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 18:49:06.431565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.431448 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xxl7k\"" Nov 03 18:49:06.431565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.431489 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 18:49:06.431565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.431500 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 18:49:06.511234 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511195 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c659d570-af36-4aca-9532-5dc9e24ef622-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.511432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511241 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lqf\" (UniqueName: \"kubernetes.io/projected/c659d570-af36-4aca-9532-5dc9e24ef622-kube-api-access-h6lqf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.511432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511282 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.511432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511334 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.511432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511370 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.511432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511398 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.511888 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511853 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.512027 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511869 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.512027 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.511930 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.513715 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.513680 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c659d570-af36-4aca-9532-5dc9e24ef622-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.513822 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.513792 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c659d570-af36-4aca-9532-5dc9e24ef622-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.573465 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.573391 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lqf\" (UniqueName: \"kubernetes.io/projected/c659d570-af36-4aca-9532-5dc9e24ef622-kube-api-access-h6lqf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ktxrg\" (UID: \"c659d570-af36-4aca-9532-5dc9e24ef622\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:06.633782 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:06.633753 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:07.013504 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:07.013477 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg"] Nov 03 18:49:07.014903 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:49:07.014876 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc659d570_af36_4aca_9532_5dc9e24ef622.slice/crio-17c7033951dcc0bf18d3c381416cf844ef5819a94393c20065350c6f897a786e WatchSource:0}: Error finding container 17c7033951dcc0bf18d3c381416cf844ef5819a94393c20065350c6f897a786e: Status 404 returned error can't find the container with id 17c7033951dcc0bf18d3c381416cf844ef5819a94393c20065350c6f897a786e Nov 03 18:49:07.598278 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:07.598238 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" event={"ID":"c659d570-af36-4aca-9532-5dc9e24ef622","Type":"ContainerStarted","Data":"17c7033951dcc0bf18d3c381416cf844ef5819a94393c20065350c6f897a786e"} Nov 03 18:49:09.611873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:09.611831 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" event={"ID":"c659d570-af36-4aca-9532-5dc9e24ef622","Type":"ContainerStarted","Data":"656f82237ef19d39b18a01de4e070a9b47d2d7df9a4ef51e8d6393760b8b7ee1"} Nov 03 18:49:09.634336 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:09.634306 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:09.732674 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:09.732619 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" podStartSLOduration=2.1408225180000002 podStartE2EDuration="3.732605879s" totalImagesPullingTime="1.591783361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:09.731059214 +0000 UTC m=+679.214055130" watchObservedRunningTime="2026-11-03 18:49:09.732605879 +0000 UTC m=+679.215601793" Nov 03 18:49:14.363659 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.363624 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf"] Nov 03 18:49:14.367304 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.367288 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.471376 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.471346 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 18:49:14.473927 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.473906 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf"] Nov 03 18:49:14.485020 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.484992 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ec304-b99e-4a18-b966-dd7e1e0c73b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.485020 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.485021 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptb6h\" (UniqueName: \"kubernetes.io/projected/621ec304-b99e-4a18-b966-dd7e1e0c73b3-kube-api-access-ptb6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.485181 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.485046 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.485181 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.485097 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.485254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.485191 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.485254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.485230 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586088 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586051 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ec304-b99e-4a18-b966-dd7e1e0c73b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586088 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586088 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptb6h\" (UniqueName: \"kubernetes.io/projected/621ec304-b99e-4a18-b966-dd7e1e0c73b3-kube-api-access-ptb6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586116 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586141 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586187 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586238 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586644 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586621 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586624 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.586753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.586679 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.588483 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.588466 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ec304-b99e-4a18-b966-dd7e1e0c73b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.588667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.588648 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ec304-b99e-4a18-b966-dd7e1e0c73b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.649248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.649184 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptb6h\" (UniqueName: \"kubernetes.io/projected/621ec304-b99e-4a18-b966-dd7e1e0c73b3-kube-api-access-ptb6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf\" (UID: \"621ec304-b99e-4a18-b966-dd7e1e0c73b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:14.680052 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:14.680028 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:15.068967 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:15.068934 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf"] Nov 03 18:49:15.069460 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:49:15.069429 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621ec304_b99e_4a18_b966_dd7e1e0c73b3.slice/crio-19bc0a681916ee7163013b8188b7d2013a387cf0d18c6941f5447316e4000367 WatchSource:0}: Error finding container 19bc0a681916ee7163013b8188b7d2013a387cf0d18c6941f5447316e4000367: Status 404 returned error can't find the container with id 19bc0a681916ee7163013b8188b7d2013a387cf0d18c6941f5447316e4000367 Nov 03 18:49:15.645768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:15.645691 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" event={"ID":"621ec304-b99e-4a18-b966-dd7e1e0c73b3","Type":"ContainerStarted","Data":"19bc0a681916ee7163013b8188b7d2013a387cf0d18c6941f5447316e4000367"} Nov 03 18:49:16.634229 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:16.634193 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:16.646790 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:16.646764 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:16.652883 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:16.652850 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" event={"ID":"621ec304-b99e-4a18-b966-dd7e1e0c73b3","Type":"ContainerStarted","Data":"73717f18bf7af4ae37e711552d7443a5ff9bd76bfba9203c85f3d7a2630e256d"} Nov 03 18:49:16.664607 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:16.664582 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" Nov 03 18:49:16.680684 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:16.680662 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:16.887180 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:16.887082 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" podStartSLOduration=2.16157477 podStartE2EDuration="2.887068084s" totalImagesPullingTime="725.493314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:16.884761056 +0000 UTC m=+686.367756973" watchObservedRunningTime="2026-11-03 18:49:16.887068084 +0000 UTC m=+686.370063997" Nov 03 18:49:24.680597 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:24.680554 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:24.693205 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:24.693181 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:24.713301 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:24.713275 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" Nov 03 18:49:24.985247 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:24.985213 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6"] Nov 03 18:49:24.989064 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:24.989040 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.067295 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.067262 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6"] Nov 03 18:49:25.090895 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.090864 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.090895 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.090893 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.091100 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.090913 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.091100 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.091064 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.091100 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.091082 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.091195 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.091161 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.091235 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.091197 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.091235 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.091216 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64ml\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-kube-api-access-h64ml\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.091311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.091296 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192124 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192092 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192124 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192126 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64ml\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-kube-api-access-h64ml\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192146 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192174 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192190 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192335 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192564 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192442 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192564 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192473 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192564 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192544 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192757 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192732 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192750 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.192924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.192901 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.193023 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.193002 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.193252 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.193232 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.194774 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.194746 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.194929 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.194914 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.256288 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.256226 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.256405 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.256386 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64ml\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-kube-api-access-h64ml\") pod \"maas-default-gateway-openshift-default-7854949c9-5rsd6\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.306379 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.306347 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:25.738912 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.737534 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6"] Nov 03 18:49:25.744674 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.744634 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:49:25.744782 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.744761 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:49:25.744831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:25.744814 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:49:26.718313 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:26.718263 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" event={"ID":"5c7c3958-42f9-4047-a05a-4bed04b4b34b","Type":"ContainerStarted","Data":"0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26"} Nov 03 18:49:26.718506 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:26.718353 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" event={"ID":"5c7c3958-42f9-4047-a05a-4bed04b4b34b","Type":"ContainerStarted","Data":"81cb371bcb83e93afa3f4ec507c15f90e81d0a1e7f675e1a47512abc61c0a19c"} Nov 03 18:49:27.306567 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:27.306526 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:27.308225 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:27.308198 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:49:27.308369 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:27.308257 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 18:49:28.307690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:28.307646 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:49:28.308204 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:28.307743 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 18:49:29.306813 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:29.306778 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:49:29.306998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:29.306828 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 18:49:30.306971 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:30.306935 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:49:30.307368 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:30.307000 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 18:49:31.307479 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:31.307438 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:49:31.307930 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:31.307506 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 18:49:32.307116 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:32.307075 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:49:32.307327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:32.307138 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 18:49:32.801587 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:49:32.801551 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:49:34.307522 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:34.307481 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:49:34.308032 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:34.307553 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:49:35.348893 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:35.348851 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-5rsd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:49:35.349324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:35.348931 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:49:36.129127 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:36.129092 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:36.129536 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:36.129517 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:36.144900 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:36.144879 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:49:36.302498 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:36.302443 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podStartSLOduration=12.302430608 podStartE2EDuration="12.302430608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:26.955003214 +0000 UTC m=+696.437999130" watchObservedRunningTime="2026-11-03 18:49:36.302430608 +0000 UTC m=+705.785426591" Nov 03 18:49:55.375014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.374965 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-74wsq"] Nov 03 18:49:55.380117 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.380097 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:55.433863 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.433836 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-74wsq"] Nov 03 18:49:55.481371 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.481342 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b24ca5fe-6e36-4175-984a-da066ff80f00-metrics-certs\") pod \"ai-gateway-controller-74f5787476-74wsq\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:55.481540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.481381 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4zn\" (UniqueName: \"kubernetes.io/projected/b24ca5fe-6e36-4175-984a-da066ff80f00-kube-api-access-zb4zn\") pod \"ai-gateway-controller-74f5787476-74wsq\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:55.482162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.482140 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 18:49:55.493248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.493192 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-2tf8f\"" Nov 03 18:49:55.493565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.493544 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 18:49:55.582308 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.582275 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b24ca5fe-6e36-4175-984a-da066ff80f00-metrics-certs\") pod \"ai-gateway-controller-74f5787476-74wsq\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:55.582308 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.582310 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4zn\" (UniqueName: \"kubernetes.io/projected/b24ca5fe-6e36-4175-984a-da066ff80f00-kube-api-access-zb4zn\") pod \"ai-gateway-controller-74f5787476-74wsq\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:55.584748 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.584725 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b24ca5fe-6e36-4175-984a-da066ff80f00-metrics-certs\") pod \"ai-gateway-controller-74f5787476-74wsq\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:55.644317 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.644249 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4zn\" (UniqueName: \"kubernetes.io/projected/b24ca5fe-6e36-4175-984a-da066ff80f00-kube-api-access-zb4zn\") pod \"ai-gateway-controller-74f5787476-74wsq\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:55.693611 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:55.693582 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:49:56.069595 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:56.069568 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-74wsq"] Nov 03 18:49:56.071484 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:49:56.071443 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24ca5fe_6e36_4175_984a_da066ff80f00.slice/crio-b2b4cd3d1abfe839211b6d004d7666ea2f0173467bbcebe4fcf5ec405126da5a WatchSource:0}: Error finding container b2b4cd3d1abfe839211b6d004d7666ea2f0173467bbcebe4fcf5ec405126da5a: Status 404 returned error can't find the container with id b2b4cd3d1abfe839211b6d004d7666ea2f0173467bbcebe4fcf5ec405126da5a Nov 03 18:49:56.888137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:49:56.888098 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" event={"ID":"b24ca5fe-6e36-4175-984a-da066ff80f00","Type":"ContainerStarted","Data":"b2b4cd3d1abfe839211b6d004d7666ea2f0173467bbcebe4fcf5ec405126da5a"} Nov 03 18:50:02.793095 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:50:02.793057 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:50:02.928137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:02.928099 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" event={"ID":"b24ca5fe-6e36-4175-984a-da066ff80f00","Type":"ContainerStarted","Data":"551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee"} Nov 03 18:50:02.928334 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:02.928184 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:50:03.049275 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:03.049174 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" podStartSLOduration=1.773941215 podStartE2EDuration="8.049160673s" totalImagesPullingTime="6.275219458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:03.047262044 +0000 UTC m=+732.530257958" watchObservedRunningTime="2026-11-03 18:50:03.049160673 +0000 UTC m=+732.532156587" Nov 03 18:50:06.805693 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.805656 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh"] Nov 03 18:50:06.809334 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.809318 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:06.863684 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.863657 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 18:50:06.866725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.866663 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh"] Nov 03 18:50:06.896560 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.896527 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgd8\" (UniqueName: \"kubernetes.io/projected/1252e5c7-6c27-4493-8cc3-f1070ba4d5a1-kube-api-access-8hgd8\") pod \"payload-pre-processing-7b98bb678c-zrpdh\" (UID: \"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:06.896790 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.896587 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252e5c7-6c27-4493-8cc3-f1070ba4d5a1-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zrpdh\" (UID: \"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:06.913866 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.913836 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mrn77\"" Nov 03 18:50:06.932321 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.932289 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kt9k6"] Nov 03 18:50:06.936758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.936734 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:06.989364 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.989322 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kt9k6"] Nov 03 18:50:06.997903 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.997872 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgd8\" (UniqueName: \"kubernetes.io/projected/1252e5c7-6c27-4493-8cc3-f1070ba4d5a1-kube-api-access-8hgd8\") pod \"payload-pre-processing-7b98bb678c-zrpdh\" (UID: \"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:06.998044 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.997918 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252e5c7-6c27-4493-8cc3-f1070ba4d5a1-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zrpdh\" (UID: \"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:06.998667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:06.998641 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252e5c7-6c27-4493-8cc3-f1070ba4d5a1-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zrpdh\" (UID: \"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:07.065165 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.065096 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgd8\" (UniqueName: \"kubernetes.io/projected/1252e5c7-6c27-4493-8cc3-f1070ba4d5a1-kube-api-access-8hgd8\") pod \"payload-pre-processing-7b98bb678c-zrpdh\" (UID: \"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:07.099085 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.099054 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a4c748-6a4e-46eb-8080-82f904149747-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-kt9k6\" (UID: \"53a4c748-6a4e-46eb-8080-82f904149747\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:07.099085 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.099089 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz87r\" (UniqueName: \"kubernetes.io/projected/53a4c748-6a4e-46eb-8080-82f904149747-kube-api-access-vz87r\") pod \"payload-processing-55fbbbf9dc-kt9k6\" (UID: \"53a4c748-6a4e-46eb-8080-82f904149747\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:07.122148 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.122120 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:07.200359 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.200327 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a4c748-6a4e-46eb-8080-82f904149747-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-kt9k6\" (UID: \"53a4c748-6a4e-46eb-8080-82f904149747\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:07.200485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.200366 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz87r\" (UniqueName: \"kubernetes.io/projected/53a4c748-6a4e-46eb-8080-82f904149747-kube-api-access-vz87r\") pod \"payload-processing-55fbbbf9dc-kt9k6\" (UID: \"53a4c748-6a4e-46eb-8080-82f904149747\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:07.201391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.201367 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a4c748-6a4e-46eb-8080-82f904149747-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-kt9k6\" (UID: \"53a4c748-6a4e-46eb-8080-82f904149747\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:07.262382 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.262350 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz87r\" (UniqueName: \"kubernetes.io/projected/53a4c748-6a4e-46eb-8080-82f904149747-kube-api-access-vz87r\") pod \"payload-processing-55fbbbf9dc-kt9k6\" (UID: \"53a4c748-6a4e-46eb-8080-82f904149747\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:07.504076 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.504037 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh"] Nov 03 18:50:07.505947 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:50:07.505921 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1252e5c7_6c27_4493_8cc3_f1070ba4d5a1.slice/crio-455519df6728d2fe1a7938cd89f00adf1146bd2fb4d3f6e5e6e2bf859105c9b7 WatchSource:0}: Error finding container 455519df6728d2fe1a7938cd89f00adf1146bd2fb4d3f6e5e6e2bf859105c9b7: Status 404 returned error can't find the container with id 455519df6728d2fe1a7938cd89f00adf1146bd2fb4d3f6e5e6e2bf859105c9b7 Nov 03 18:50:07.551840 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.551804 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:07.930543 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.930515 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kt9k6"] Nov 03 18:50:07.931561 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:50:07.931530 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a4c748_6a4e_46eb_8080_82f904149747.slice/crio-00b65df594c7828de4a68f2c6d4bdff61bd233dd7bec89804e9b1432526eddb5 WatchSource:0}: Error finding container 00b65df594c7828de4a68f2c6d4bdff61bd233dd7bec89804e9b1432526eddb5: Status 404 returned error can't find the container with id 00b65df594c7828de4a68f2c6d4bdff61bd233dd7bec89804e9b1432526eddb5 Nov 03 18:50:07.958476 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.958444 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" event={"ID":"53a4c748-6a4e-46eb-8080-82f904149747","Type":"ContainerStarted","Data":"00b65df594c7828de4a68f2c6d4bdff61bd233dd7bec89804e9b1432526eddb5"} Nov 03 18:50:07.959667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:07.959644 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" event={"ID":"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1","Type":"ContainerStarted","Data":"455519df6728d2fe1a7938cd89f00adf1146bd2fb4d3f6e5e6e2bf859105c9b7"} Nov 03 18:50:10.983853 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:10.983817 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" event={"ID":"53a4c748-6a4e-46eb-8080-82f904149747","Type":"ContainerStarted","Data":"14dff954ea083bb503973183ed246280b35d019c13e32870c788984453aa27c7"} Nov 03 18:50:10.984216 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:10.983893 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:10.985424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:10.985398 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" event={"ID":"1252e5c7-6c27-4493-8cc3-f1070ba4d5a1","Type":"ContainerStarted","Data":"be8e3a509e0907f4ca4c68b7f894448b082aae034af5514aa3ee3add8c1cba86"} Nov 03 18:50:10.985530 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:10.985520 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:11.102611 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:11.102564 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" podStartSLOduration=2.138052759 podStartE2EDuration="5.102551878s" totalImagesPullingTime="2.964499119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:11.102429996 +0000 UTC m=+740.585425910" watchObservedRunningTime="2026-11-03 18:50:11.102551878 +0000 UTC m=+740.585547791" Nov 03 18:50:11.231386 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:11.231326 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" podStartSLOduration=2.106390928 podStartE2EDuration="5.231309731s" totalImagesPullingTime="3.124918803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:11.22978176 +0000 UTC m=+740.712777675" watchObservedRunningTime="2026-11-03 18:50:11.231309731 +0000 UTC m=+740.714305644" Nov 03 18:50:13.936487 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:13.936453 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:50:14.302324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.302238 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f84cbd65-zgqhf"] Nov 03 18:50:14.306598 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.306574 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:14.362998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.362963 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f84cbd65-zgqhf"] Nov 03 18:50:14.480562 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.480522 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cgv\" (UniqueName: \"kubernetes.io/projected/444955b0-fba7-45b4-b7ab-bb94095f64df-kube-api-access-95cgv\") pod \"ai-gateway-controller-5f84cbd65-zgqhf\" (UID: \"444955b0-fba7-45b4-b7ab-bb94095f64df\") " pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:14.480763 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.480744 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444955b0-fba7-45b4-b7ab-bb94095f64df-metrics-certs\") pod \"ai-gateway-controller-5f84cbd65-zgqhf\" (UID: \"444955b0-fba7-45b4-b7ab-bb94095f64df\") " pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:14.581615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.581534 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444955b0-fba7-45b4-b7ab-bb94095f64df-metrics-certs\") pod \"ai-gateway-controller-5f84cbd65-zgqhf\" (UID: \"444955b0-fba7-45b4-b7ab-bb94095f64df\") " pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:14.581615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.581583 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cgv\" (UniqueName: \"kubernetes.io/projected/444955b0-fba7-45b4-b7ab-bb94095f64df-kube-api-access-95cgv\") pod \"ai-gateway-controller-5f84cbd65-zgqhf\" (UID: \"444955b0-fba7-45b4-b7ab-bb94095f64df\") " pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:14.584032 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.584008 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444955b0-fba7-45b4-b7ab-bb94095f64df-metrics-certs\") pod \"ai-gateway-controller-5f84cbd65-zgqhf\" (UID: \"444955b0-fba7-45b4-b7ab-bb94095f64df\") " pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:14.644022 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.643987 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cgv\" (UniqueName: \"kubernetes.io/projected/444955b0-fba7-45b4-b7ab-bb94095f64df-kube-api-access-95cgv\") pod \"ai-gateway-controller-5f84cbd65-zgqhf\" (UID: \"444955b0-fba7-45b4-b7ab-bb94095f64df\") " pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:14.922292 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:14.922207 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:15.303441 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:50:15.303410 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444955b0_fba7_45b4_b7ab_bb94095f64df.slice/crio-5fd34c103db71d532df69930cb5a4ae3076bdd2a00b4089a4e7c6bedb226f595 WatchSource:0}: Error finding container 5fd34c103db71d532df69930cb5a4ae3076bdd2a00b4089a4e7c6bedb226f595: Status 404 returned error can't find the container with id 5fd34c103db71d532df69930cb5a4ae3076bdd2a00b4089a4e7c6bedb226f595 Nov 03 18:50:15.303794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:15.303643 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f84cbd65-zgqhf"] Nov 03 18:50:16.014322 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:16.014288 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" event={"ID":"444955b0-fba7-45b4-b7ab-bb94095f64df","Type":"ContainerStarted","Data":"5fd34c103db71d532df69930cb5a4ae3076bdd2a00b4089a4e7c6bedb226f595"} Nov 03 18:50:17.021414 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:17.021378 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" event={"ID":"444955b0-fba7-45b4-b7ab-bb94095f64df","Type":"ContainerStarted","Data":"34f52618ce6e00ffdd1fa048098234e8fa9772fa127d53ee6743379af16696c3"} Nov 03 18:50:17.021833 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:17.021513 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:17.142627 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:17.142564 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" podStartSLOduration=2.360531991 podStartE2EDuration="3.142545187s" totalImagesPullingTime="782.013196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:17.140886199 +0000 UTC m=+746.623882117" watchObservedRunningTime="2026-11-03 18:50:17.142545187 +0000 UTC m=+746.625541101" Nov 03 18:50:21.993638 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:21.993605 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zrpdh" Nov 03 18:50:21.994071 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:21.993675 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-kt9k6" Nov 03 18:50:28.029017 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.028985 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f84cbd65-zgqhf" Nov 03 18:50:28.174939 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.174907 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-74wsq"] Nov 03 18:50:28.175970 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.175883 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" podUID="b24ca5fe-6e36-4175-984a-da066ff80f00" containerName="manager" containerID="cri-o://551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee" gracePeriod=30 Nov 03 18:50:28.433343 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.433318 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:50:28.497831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.497803 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24ca5fe-6e36-4175-984a-da066ff80f00-kube-api-access-zb4zn\" (UniqueName: \"kubernetes.io/projected/b24ca5fe-6e36-4175-984a-da066ff80f00-kube-api-access-zb4zn\") pod \"b24ca5fe-6e36-4175-984a-da066ff80f00\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " Nov 03 18:50:28.499780 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.499753 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24ca5fe-6e36-4175-984a-da066ff80f00-kube-api-access-zb4zn" pod "b24ca5fe-6e36-4175-984a-da066ff80f00" (UID: "b24ca5fe-6e36-4175-984a-da066ff80f00"). InnerVolumeSpecName "kube-api-access-zb4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:50:28.598999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.598921 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24ca5fe-6e36-4175-984a-da066ff80f00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b24ca5fe-6e36-4175-984a-da066ff80f00-metrics-certs\") pod \"b24ca5fe-6e36-4175-984a-da066ff80f00\" (UID: \"b24ca5fe-6e36-4175-984a-da066ff80f00\") " Nov 03 18:50:28.599187 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.599170 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb4zn\" (UniqueName: \"kubernetes.io/projected/b24ca5fe-6e36-4175-984a-da066ff80f00-kube-api-access-zb4zn\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:28.600970 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.600936 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24ca5fe-6e36-4175-984a-da066ff80f00-metrics-certs" pod "b24ca5fe-6e36-4175-984a-da066ff80f00" (UID: "b24ca5fe-6e36-4175-984a-da066ff80f00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:50:28.700461 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:28.700430 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b24ca5fe-6e36-4175-984a-da066ff80f00-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:50:29.090134 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.090102 4948 generic.go:356] "Generic (PLEG): container finished" podID="b24ca5fe-6e36-4175-984a-da066ff80f00" containerID="551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee" exitCode=0 Nov 03 18:50:29.090546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.090152 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" event={"ID":"b24ca5fe-6e36-4175-984a-da066ff80f00","Type":"ContainerDied","Data":"551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee"} Nov 03 18:50:29.090546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.090161 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" Nov 03 18:50:29.090546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.090191 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-74wsq" event={"ID":"b24ca5fe-6e36-4175-984a-da066ff80f00","Type":"ContainerDied","Data":"b2b4cd3d1abfe839211b6d004d7666ea2f0173467bbcebe4fcf5ec405126da5a"} Nov 03 18:50:29.090546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.090213 4948 scope.go:122] "RemoveContainer" containerID="551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee" Nov 03 18:50:29.100108 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.100086 4948 scope.go:122] "RemoveContainer" containerID="551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee" Nov 03 18:50:29.100415 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:50:29.100393 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee\": container with ID starting with 551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee not found: ID does not exist" containerID="551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee" Nov 03 18:50:29.100482 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.100428 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee"} err="failed to get container status \"551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee\": rpc error: code = NotFound desc = could not find container \"551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee\": container with ID starting with 551289373e94574b5640e0116d6a3a0ae18f7277198f8914bfda2c04ea6d19ee not found: ID does not exist" Nov 03 18:50:29.257611 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.257577 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-74wsq"] Nov 03 18:50:29.261745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:29.261717 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-74wsq"] Nov 03 18:50:30.752108 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:30.752033 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24ca5fe-6e36-4175-984a-da066ff80f00" path="/var/lib/kubelet/pods/b24ca5fe-6e36-4175-984a-da066ff80f00/volumes" Nov 03 18:50:32.792849 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:50:32.792809 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:50:58.597581 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.597540 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5"] Nov 03 18:50:58.598084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.598008 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b24ca5fe-6e36-4175-984a-da066ff80f00" containerName="manager" Nov 03 18:50:58.598084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.598021 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24ca5fe-6e36-4175-984a-da066ff80f00" containerName="manager" Nov 03 18:50:58.598155 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.598112 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24ca5fe-6e36-4175-984a-da066ff80f00" containerName="manager" Nov 03 18:50:58.603089 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.603069 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.658065 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.658025 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fba266-openshift-default-dockercfg-dm45z\"" Nov 03 18:50:58.674662 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.674630 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.674876 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.674672 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.674876 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.674712 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.674876 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.674835 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.675129 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.674901 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.675129 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.674937 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.675129 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.675003 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.675129 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.675022 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrclq\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-kube-api-access-rrclq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.675129 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.675045 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.710266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.710219 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5"] Nov 03 18:50:58.776351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776314 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776367 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrclq\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-kube-api-access-rrclq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776390 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776435 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776460 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776501 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776527 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776571 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776629 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776739 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.776830 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.776824 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.777058 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.777015 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.777116 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.777092 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.777245 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.777229 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.778844 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.778825 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.779001 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.778985 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.841011 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.840983 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.841251 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.841226 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrclq\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-kube-api-access-rrclq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.884455 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.884393 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5"] Nov 03 18:50:58.887801 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.887779 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.915785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.915754 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:50:58.947647 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.947620 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5"] Nov 03 18:50:58.950889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.950866 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2e052a-openshift-default-dockercfg-vjkcv\"" Nov 03 18:50:58.979164 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979080 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979164 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979127 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979309 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979165 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979309 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979197 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979309 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979254 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979309 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979281 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979520 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979308 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6r4t\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-kube-api-access-z6r4t\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979520 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979360 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:58.979520 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:58.979397 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080455 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080425 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080455 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080459 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080483 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080505 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080540 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080557 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080573 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6r4t\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-kube-api-access-z6r4t\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080984 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080743 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080984 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080805 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080984 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080945 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.080984 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.080969 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.081195 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.081012 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.081195 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.081164 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.081412 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.081390 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.082929 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.082902 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.083426 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.083406 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.161335 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.161269 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.163872 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.163847 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6r4t\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-kube-api-access-z6r4t\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.199960 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.199925 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:50:59.354839 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.354811 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5"] Nov 03 18:50:59.354974 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:50:59.354946 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0eeab63_2d7c_4cb5_9b2c_1135c47d924a.slice/crio-faaa3381cb33ddbcdf0d86767b23f2e389db72d38cdb9fcf4f1bb24329bc956e WatchSource:0}: Error finding container faaa3381cb33ddbcdf0d86767b23f2e389db72d38cdb9fcf4f1bb24329bc956e: Status 404 returned error can't find the container with id faaa3381cb33ddbcdf0d86767b23f2e389db72d38cdb9fcf4f1bb24329bc956e Nov 03 18:50:59.357188 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.357156 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:50:59.357297 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.357233 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:50:59.357297 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.357282 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:50:59.605873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.605804 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5"] Nov 03 18:50:59.612675 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:50:59.612650 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f639da3_66ce_4c7b_b94a_5f9edbf6651c.slice/crio-52e647fd85bd7bf0ce9c920b20c58da4aa46f5301321b4b7c36e335d4a4d8c46 WatchSource:0}: Error finding container 52e647fd85bd7bf0ce9c920b20c58da4aa46f5301321b4b7c36e335d4a4d8c46: Status 404 returned error can't find the container with id 52e647fd85bd7bf0ce9c920b20c58da4aa46f5301321b4b7c36e335d4a4d8c46 Nov 03 18:50:59.615766 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.615684 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:50:59.615848 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.615813 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:50:59.615886 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.615857 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:50:59.831797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.831763 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd"] Nov 03 18:50:59.834466 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.834445 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.892432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.891402 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9686c3-openshift-default-dockercfg-kfklz\"" Nov 03 18:50:59.895267 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.895242 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd"] Nov 03 18:50:59.997258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997219 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997467 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997284 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997467 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997341 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997467 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997390 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m99pz\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-kube-api-access-m99pz\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997467 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997438 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997688 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997511 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997688 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997544 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997688 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997612 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:50:59.997688 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:50:59.997673 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098351 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098311 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m99pz\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-kube-api-access-m99pz\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098359 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098530 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098663 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098576 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098663 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098626 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098813 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098676 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098813 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098736 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098813 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098765 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.098813 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.098799 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.099033 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.099007 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.099141 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.099118 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.099199 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.099178 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.099250 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.099230 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.099370 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.099353 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.101032 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.101010 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.101382 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.101361 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.162091 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.162023 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.162493 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.162463 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m99pz\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-kube-api-access-m99pz\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.162758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.162736 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4"] Nov 03 18:51:00.166066 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.166046 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.225145 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.224933 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-32a59b-openshift-default-dockercfg-rm7v5\"" Nov 03 18:51:00.228311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.228279 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4"] Nov 03 18:51:00.273353 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.273315 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" event={"ID":"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a","Type":"ContainerStarted","Data":"8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81"} Nov 03 18:51:00.273526 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.273378 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" event={"ID":"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a","Type":"ContainerStarted","Data":"faaa3381cb33ddbcdf0d86767b23f2e389db72d38cdb9fcf4f1bb24329bc956e"} Nov 03 18:51:00.275408 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.275384 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" event={"ID":"4f639da3-66ce-4c7b-b94a-5f9edbf6651c","Type":"ContainerStarted","Data":"70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90"} Nov 03 18:51:00.275526 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.275422 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" event={"ID":"4f639da3-66ce-4c7b-b94a-5f9edbf6651c","Type":"ContainerStarted","Data":"52e647fd85bd7bf0ce9c920b20c58da4aa46f5301321b4b7c36e335d4a4d8c46"} Nov 03 18:51:00.300851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.300817 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.300851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.300854 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.301080 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.300876 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.301080 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.300954 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxjj\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-kube-api-access-cwxjj\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.301080 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.301002 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.301080 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.301060 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.301286 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.301086 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.301286 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.301107 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.301286 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.301168 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402280 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402243 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402298 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402344 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402370 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxjj\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-kube-api-access-cwxjj\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402429 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402597 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402645 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402676 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402682 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402938 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402754 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.402999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.402972 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.403057 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.403020 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.403287 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.403263 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.403521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.403502 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.404912 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.404887 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.405166 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.405149 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.414780 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.414687 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podStartSLOduration=2.414674507 podStartE2EDuration="2.414674507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:00.413057152 +0000 UTC m=+789.896053066" watchObservedRunningTime="2026-11-03 18:51:00.414674507 +0000 UTC m=+789.897670425" Nov 03 18:51:00.447745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.447707 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:00.464612 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.464580 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.464976 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.464953 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxjj\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-kube-api-access-cwxjj\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.498798 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.498761 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:00.536632 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.536586 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podStartSLOduration=2.536572849 podStartE2EDuration="2.536572849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:00.535148728 +0000 UTC m=+790.018144679" watchObservedRunningTime="2026-11-03 18:51:00.536572849 +0000 UTC m=+790.019568845" Nov 03 18:51:00.833463 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.833426 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd"] Nov 03 18:51:00.834859 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:00.834837 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd255599_9154_4ee4_b1bf_3b44fb198f9a.slice/crio-5da8e0fdc7d899c9dc7b3097ea7a16c1637e07a51366e50ab2c1fc061b356909 WatchSource:0}: Error finding container 5da8e0fdc7d899c9dc7b3097ea7a16c1637e07a51366e50ab2c1fc061b356909: Status 404 returned error can't find the container with id 5da8e0fdc7d899c9dc7b3097ea7a16c1637e07a51366e50ab2c1fc061b356909 Nov 03 18:51:00.844936 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.844822 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:00.845030 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.845002 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:00.845072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.845058 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:00.916769 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.916737 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:51:00.922637 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.922615 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:51:00.927879 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.927854 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4"] Nov 03 18:51:00.929115 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:00.929084 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d25ebd_3ae1_409d_9f84_70bb798ba6a8.slice/crio-196aa90b76c1eedf82c5a936aa5766fcabc657e6c574c2450dd3863d65ddf985 WatchSource:0}: Error finding container 196aa90b76c1eedf82c5a936aa5766fcabc657e6c574c2450dd3863d65ddf985: Status 404 returned error can't find the container with id 196aa90b76c1eedf82c5a936aa5766fcabc657e6c574c2450dd3863d65ddf985 Nov 03 18:51:00.931819 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.931777 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:00.931926 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.931864 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:00.931926 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:00.931903 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:01.200173 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.200135 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:01.205081 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.205054 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:01.301979 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.299561 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" event={"ID":"26d25ebd-3ae1-409d-9f84-70bb798ba6a8","Type":"ContainerStarted","Data":"490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc"} Nov 03 18:51:01.301979 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.299629 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" event={"ID":"26d25ebd-3ae1-409d-9f84-70bb798ba6a8","Type":"ContainerStarted","Data":"196aa90b76c1eedf82c5a936aa5766fcabc657e6c574c2450dd3863d65ddf985"} Nov 03 18:51:01.304732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.303226 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" event={"ID":"dd255599-9154-4ee4-b1bf-3b44fb198f9a","Type":"ContainerStarted","Data":"2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691"} Nov 03 18:51:01.304732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.303812 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:51:01.304732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.303851 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" event={"ID":"dd255599-9154-4ee4-b1bf-3b44fb198f9a","Type":"ContainerStarted","Data":"5da8e0fdc7d899c9dc7b3097ea7a16c1637e07a51366e50ab2c1fc061b356909"} Nov 03 18:51:01.304732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.303905 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:01.305055 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.304940 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:01.305245 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.305229 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:51:01.448854 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.448818 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:01.455152 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.455049 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:01.499145 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.499114 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:01.576519 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.576463 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podStartSLOduration=1.576449491 podStartE2EDuration="1.576449491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:01.574650233 +0000 UTC m=+791.057646158" watchObservedRunningTime="2026-11-03 18:51:01.576449491 +0000 UTC m=+791.059445404" Nov 03 18:51:01.705028 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:01.702745 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" podStartSLOduration=2.702691917 podStartE2EDuration="2.702691917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:01.702529802 +0000 UTC m=+791.185525699" watchObservedRunningTime="2026-11-03 18:51:01.702691917 +0000 UTC m=+791.185687832" Nov 03 18:51:02.308158 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:02.308122 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:02.308886 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:02.308868 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:02.504866 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:02.504838 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:02.793124 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:51:02.793081 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:51:03.313627 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:03.313588 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:03.314526 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:03.314501 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:09.484464 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.484430 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2"] Nov 03 18:51:09.487248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.487200 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.509023 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.508996 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fa162-17a9-4f0a-ad54-8ee6add5071d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.509153 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.509050 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhp6z\" (UniqueName: \"kubernetes.io/projected/e47fa162-17a9-4f0a-ad54-8ee6add5071d-kube-api-access-xhp6z\") pod \"payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.542977 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.542947 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2"] Nov 03 18:51:09.616164 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.616129 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fa162-17a9-4f0a-ad54-8ee6add5071d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.616310 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.616187 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhp6z\" (UniqueName: \"kubernetes.io/projected/e47fa162-17a9-4f0a-ad54-8ee6add5071d-kube-api-access-xhp6z\") pod \"payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.616617 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.616599 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-fba266-dockercfg-58q8n\"" Nov 03 18:51:09.616676 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.616666 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-fba266\"" Nov 03 18:51:09.617463 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.617441 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fa162-17a9-4f0a-ad54-8ee6add5071d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.671256 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.671221 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f"] Nov 03 18:51:09.675424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.675401 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:09.686968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.686941 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhp6z\" (UniqueName: \"kubernetes.io/projected/e47fa162-17a9-4f0a-ad54-8ee6add5071d-kube-api-access-xhp6z\") pod \"payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.700798 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.700766 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f"] Nov 03 18:51:09.799963 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.799866 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:09.827558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.827519 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d4548b-1f30-482a-b681-92323671e3d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:09.828097 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.828066 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrq5\" (UniqueName: \"kubernetes.io/projected/92d4548b-1f30-482a-b681-92323671e3d4-kube-api-access-fbrq5\") pod \"payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:09.946755 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.942729 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrq5\" (UniqueName: \"kubernetes.io/projected/92d4548b-1f30-482a-b681-92323671e3d4-kube-api-access-fbrq5\") pod \"payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:09.946755 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.942819 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d4548b-1f30-482a-b681-92323671e3d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:09.946755 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:09.943862 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d4548b-1f30-482a-b681-92323671e3d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:10.007338 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.007307 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrq5\" (UniqueName: \"kubernetes.io/projected/92d4548b-1f30-482a-b681-92323671e3d4-kube-api-access-fbrq5\") pod \"payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:10.198992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.198963 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2"] Nov 03 18:51:10.200443 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:10.200419 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47fa162_17a9_4f0a_ad54_8ee6add5071d.slice/crio-a8cdebbb8df0d56ffc180f685af85a9667d1e6cf92c3aff10f84551b51534914 WatchSource:0}: Error finding container a8cdebbb8df0d56ffc180f685af85a9667d1e6cf92c3aff10f84551b51534914: Status 404 returned error can't find the container with id a8cdebbb8df0d56ffc180f685af85a9667d1e6cf92c3aff10f84551b51534914 Nov 03 18:51:10.288691 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.288660 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:10.380689 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.380640 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" event={"ID":"e47fa162-17a9-4f0a-ad54-8ee6add5071d","Type":"ContainerStarted","Data":"151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493"} Nov 03 18:51:10.380873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.380734 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" event={"ID":"e47fa162-17a9-4f0a-ad54-8ee6add5071d","Type":"ContainerStarted","Data":"a8cdebbb8df0d56ffc180f685af85a9667d1e6cf92c3aff10f84551b51534914"} Nov 03 18:51:10.380953 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.380908 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:10.502619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.502515 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" podStartSLOduration=1.502501247 podStartE2EDuration="1.502501247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:10.501124621 +0000 UTC m=+799.984120534" watchObservedRunningTime="2026-11-03 18:51:10.502501247 +0000 UTC m=+799.985497161" Nov 03 18:51:10.665820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:10.665794 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f"] Nov 03 18:51:10.666381 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:10.666356 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d4548b_1f30_482a_b681_92323671e3d4.slice/crio-b72cbd0a2dadb97932ba2415a7fe6a516ca143261a3a05f7b6eae8dfcf54650e WatchSource:0}: Error finding container b72cbd0a2dadb97932ba2415a7fe6a516ca143261a3a05f7b6eae8dfcf54650e: Status 404 returned error can't find the container with id b72cbd0a2dadb97932ba2415a7fe6a516ca143261a3a05f7b6eae8dfcf54650e Nov 03 18:51:11.388138 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:11.388096 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" event={"ID":"92d4548b-1f30-482a-b681-92323671e3d4","Type":"ContainerStarted","Data":"aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1"} Nov 03 18:51:11.388354 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:11.388164 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" event={"ID":"92d4548b-1f30-482a-b681-92323671e3d4","Type":"ContainerStarted","Data":"b72cbd0a2dadb97932ba2415a7fe6a516ca143261a3a05f7b6eae8dfcf54650e"} Nov 03 18:51:11.388354 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:11.388194 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:11.508718 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:11.508640 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" podStartSLOduration=2.508623931 podStartE2EDuration="2.508623931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:11.506041947 +0000 UTC m=+800.989037861" watchObservedRunningTime="2026-11-03 18:51:11.508623931 +0000 UTC m=+800.991619846" Nov 03 18:51:14.595496 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.595456 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5"] Nov 03 18:51:14.599187 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.599166 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:14.671972 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.671942 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2e052a\"" Nov 03 18:51:14.672141 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.672018 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2e052a-dockercfg-685vn\"" Nov 03 18:51:14.697676 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.697639 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041633e-2fd3-4de7-a336-c09632af88a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:14.698069 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.698041 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfw77\" (UniqueName: \"kubernetes.io/projected/0041633e-2fd3-4de7-a336-c09632af88a4-kube-api-access-hfw77\") pod \"payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:14.774850 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.774815 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5"] Nov 03 18:51:14.791118 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.791087 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5"] Nov 03 18:51:14.794117 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.794092 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:14.800620 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.800590 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfw77\" (UniqueName: \"kubernetes.io/projected/0041633e-2fd3-4de7-a336-c09632af88a4-kube-api-access-hfw77\") pod \"payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:14.800778 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.800656 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041633e-2fd3-4de7-a336-c09632af88a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:14.801608 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.801562 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041633e-2fd3-4de7-a336-c09632af88a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:14.851767 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.851666 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5"] Nov 03 18:51:14.863687 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.863654 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfw77\" (UniqueName: \"kubernetes.io/projected/0041633e-2fd3-4de7-a336-c09632af88a4-kube-api-access-hfw77\") pod \"payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:14.903745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.903623 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dd5f2c-306b-47fc-b243-4c2880adbdb9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:14.905233 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.905124 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cm7z\" (UniqueName: \"kubernetes.io/projected/11dd5f2c-306b-47fc-b243-4c2880adbdb9-kube-api-access-5cm7z\") pod \"payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:14.915322 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:14.914361 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:15.007574 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.007533 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dd5f2c-306b-47fc-b243-4c2880adbdb9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:15.007789 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.007658 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cm7z\" (UniqueName: \"kubernetes.io/projected/11dd5f2c-306b-47fc-b243-4c2880adbdb9-kube-api-access-5cm7z\") pod \"payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:15.008399 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.008374 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dd5f2c-306b-47fc-b243-4c2880adbdb9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:15.070238 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.070208 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cm7z\" (UniqueName: \"kubernetes.io/projected/11dd5f2c-306b-47fc-b243-4c2880adbdb9-kube-api-access-5cm7z\") pod \"payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:15.108070 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.107991 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:15.297347 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.297319 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5"] Nov 03 18:51:15.297450 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:15.297369 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0041633e_2fd3_4de7_a336_c09632af88a4.slice/crio-479b814abfacf21c661a59451b388307d2c2a2de726b9948250df86b102e8fd0 WatchSource:0}: Error finding container 479b814abfacf21c661a59451b388307d2c2a2de726b9948250df86b102e8fd0: Status 404 returned error can't find the container with id 479b814abfacf21c661a59451b388307d2c2a2de726b9948250df86b102e8fd0 Nov 03 18:51:15.413406 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.413311 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" event={"ID":"0041633e-2fd3-4de7-a336-c09632af88a4","Type":"ContainerStarted","Data":"fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca"} Nov 03 18:51:15.413406 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.413362 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" event={"ID":"0041633e-2fd3-4de7-a336-c09632af88a4","Type":"ContainerStarted","Data":"479b814abfacf21c661a59451b388307d2c2a2de726b9948250df86b102e8fd0"} Nov 03 18:51:15.413632 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.413505 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:15.484831 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.484801 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5"] Nov 03 18:51:15.486400 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:15.486370 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dd5f2c_306b_47fc_b243_4c2880adbdb9.slice/crio-1d72ed3026b25bbb2338c0cea135dd8f85759c9e30977d85352731efa9d96d9b WatchSource:0}: Error finding container 1d72ed3026b25bbb2338c0cea135dd8f85759c9e30977d85352731efa9d96d9b: Status 404 returned error can't find the container with id 1d72ed3026b25bbb2338c0cea135dd8f85759c9e30977d85352731efa9d96d9b Nov 03 18:51:15.534543 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:15.534499 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" podStartSLOduration=1.534478235 podStartE2EDuration="1.534478235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:15.533894712 +0000 UTC m=+805.016890620" watchObservedRunningTime="2026-11-03 18:51:15.534478235 +0000 UTC m=+805.017474148" Nov 03 18:51:16.425577 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:16.425537 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" event={"ID":"11dd5f2c-306b-47fc-b243-4c2880adbdb9","Type":"ContainerStarted","Data":"751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4"} Nov 03 18:51:16.426082 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:16.425601 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" event={"ID":"11dd5f2c-306b-47fc-b243-4c2880adbdb9","Type":"ContainerStarted","Data":"1d72ed3026b25bbb2338c0cea135dd8f85759c9e30977d85352731efa9d96d9b"} Nov 03 18:51:16.426082 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:16.425742 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:16.545805 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:16.545749 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" podStartSLOduration=2.545736228 podStartE2EDuration="2.545736228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:16.543160252 +0000 UTC m=+806.026156166" watchObservedRunningTime="2026-11-03 18:51:16.545736228 +0000 UTC m=+806.028732141" Nov 03 18:51:17.301397 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.301358 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57"] Nov 03 18:51:17.304609 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.304573 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.360122 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.360088 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57"] Nov 03 18:51:17.443616 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.431234 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a4cacd\"" Nov 03 18:51:17.443616 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.431495 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a4cacd-dockercfg-bdw8w\"" Nov 03 18:51:17.443616 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.433135 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mdq\" (UniqueName: \"kubernetes.io/projected/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-kube-api-access-t9mdq\") pod \"payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.443616 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.433184 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.484986 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.484952 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7"] Nov 03 18:51:17.487803 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.487781 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:17.508965 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.508937 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7"] Nov 03 18:51:17.534416 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.534386 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a3e420-71c3-4cca-9036-323dfd7f1eba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:17.534571 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.534440 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l49x\" (UniqueName: \"kubernetes.io/projected/98a3e420-71c3-4cca-9036-323dfd7f1eba-kube-api-access-5l49x\") pod \"payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:17.534671 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.534651 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mdq\" (UniqueName: \"kubernetes.io/projected/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-kube-api-access-t9mdq\") pod \"payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.534757 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.534737 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.535451 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.535421 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.605889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.605804 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mdq\" (UniqueName: \"kubernetes.io/projected/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-kube-api-access-t9mdq\") pod \"payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.619396 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.619365 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:17.635511 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.635482 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a3e420-71c3-4cca-9036-323dfd7f1eba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:17.635625 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.635525 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l49x\" (UniqueName: \"kubernetes.io/projected/98a3e420-71c3-4cca-9036-323dfd7f1eba-kube-api-access-5l49x\") pod \"payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:17.636217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.636195 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a3e420-71c3-4cca-9036-323dfd7f1eba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:17.698737 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.698711 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l49x\" (UniqueName: \"kubernetes.io/projected/98a3e420-71c3-4cca-9036-323dfd7f1eba-kube-api-access-5l49x\") pod \"payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:17.806045 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:17.802999 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:18.001171 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.001142 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57"] Nov 03 18:51:18.001735 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:18.001706 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd2d734_ddb8_47d5_8a3d_c149f684b6d4.slice/crio-6c25e38641426f7dc3da7091604088457fd2d6a3e57413cfc4461d89088b540b WatchSource:0}: Error finding container 6c25e38641426f7dc3da7091604088457fd2d6a3e57413cfc4461d89088b540b: Status 404 returned error can't find the container with id 6c25e38641426f7dc3da7091604088457fd2d6a3e57413cfc4461d89088b540b Nov 03 18:51:18.205983 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.205957 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7"] Nov 03 18:51:18.207008 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:18.206979 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a3e420_71c3_4cca_9036_323dfd7f1eba.slice/crio-5ef9b3adeaba74492f933a8c191a206f9f117cefda45f587b5295b7f891017cb WatchSource:0}: Error finding container 5ef9b3adeaba74492f933a8c191a206f9f117cefda45f587b5295b7f891017cb: Status 404 returned error can't find the container with id 5ef9b3adeaba74492f933a8c191a206f9f117cefda45f587b5295b7f891017cb Nov 03 18:51:18.458361 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.458270 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" event={"ID":"98a3e420-71c3-4cca-9036-323dfd7f1eba","Type":"ContainerStarted","Data":"1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d"} Nov 03 18:51:18.458361 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.458339 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" event={"ID":"98a3e420-71c3-4cca-9036-323dfd7f1eba","Type":"ContainerStarted","Data":"5ef9b3adeaba74492f933a8c191a206f9f117cefda45f587b5295b7f891017cb"} Nov 03 18:51:18.458808 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.458461 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:18.459977 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.459953 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" event={"ID":"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4","Type":"ContainerStarted","Data":"9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305"} Nov 03 18:51:18.460090 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.459991 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" event={"ID":"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4","Type":"ContainerStarted","Data":"6c25e38641426f7dc3da7091604088457fd2d6a3e57413cfc4461d89088b540b"} Nov 03 18:51:18.460090 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.460023 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:18.576399 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.576355 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" podStartSLOduration=1.576344159 podStartE2EDuration="1.576344159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:18.575690925 +0000 UTC m=+808.058686839" watchObservedRunningTime="2026-11-03 18:51:18.576344159 +0000 UTC m=+808.059340073" Nov 03 18:51:18.693455 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:18.693404 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" podStartSLOduration=1.6933904800000001 podStartE2EDuration="1.69339048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:18.692763605 +0000 UTC m=+808.175759540" watchObservedRunningTime="2026-11-03 18:51:18.69339048 +0000 UTC m=+808.176386394" Nov 03 18:51:19.888991 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:19.888947 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt"] Nov 03 18:51:19.892856 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:19.892830 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:19.945569 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:19.945535 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt"] Nov 03 18:51:19.983276 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:19.983230 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e06228-9483-46d1-937c-5b81dda76fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:19.983486 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:19.983297 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rk7\" (UniqueName: \"kubernetes.io/projected/c9e06228-9483-46d1-937c-5b81dda76fb4-kube-api-access-r5rk7\") pod \"payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:19.992594 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:19.992549 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-355e1a\"" Nov 03 18:51:19.992775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:19.992661 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-355e1a-dockercfg-b8ztx\"" Nov 03 18:51:20.100927 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:20.084336 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rk7\" (UniqueName: \"kubernetes.io/projected/c9e06228-9483-46d1-937c-5b81dda76fb4-kube-api-access-r5rk7\") pod \"payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:20.101425 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:20.101386 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e06228-9483-46d1-937c-5b81dda76fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:20.136724 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:20.102519 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e06228-9483-46d1-937c-5b81dda76fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:20.177928 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:20.177853 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rk7\" (UniqueName: \"kubernetes.io/projected/c9e06228-9483-46d1-937c-5b81dda76fb4-kube-api-access-r5rk7\") pod \"payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:20.205979 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:20.205948 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:20.604628 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:20.604598 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt"] Nov 03 18:51:20.605197 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:20.605166 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e06228_9483_46d1_937c_5b81dda76fb4.slice/crio-7c36ff5bd27d27295712255721a35cd7c97f0278725547ae30a37bb0df11ed9c WatchSource:0}: Error finding container 7c36ff5bd27d27295712255721a35cd7c97f0278725547ae30a37bb0df11ed9c: Status 404 returned error can't find the container with id 7c36ff5bd27d27295712255721a35cd7c97f0278725547ae30a37bb0df11ed9c Nov 03 18:51:21.391588 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:21.391553 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 18:51:21.496878 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:21.496838 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" event={"ID":"c9e06228-9483-46d1-937c-5b81dda76fb4","Type":"ContainerStarted","Data":"5cb78c0770d2fe98732e2edddb1d6d56759826954d25205064c3de239ba6171c"} Nov 03 18:51:21.497051 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:21.496889 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" event={"ID":"c9e06228-9483-46d1-937c-5b81dda76fb4","Type":"ContainerStarted","Data":"7c36ff5bd27d27295712255721a35cd7c97f0278725547ae30a37bb0df11ed9c"} Nov 03 18:51:21.497051 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:21.496921 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:21.630602 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:21.630541 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" podStartSLOduration=2.630522251 podStartE2EDuration="2.630522251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:21.627156891 +0000 UTC m=+811.110152802" watchObservedRunningTime="2026-11-03 18:51:21.630522251 +0000 UTC m=+811.113518166" Nov 03 18:51:22.397844 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.397816 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 18:51:22.579845 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.579800 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z"] Nov 03 18:51:22.582882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.582854 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:22.633356 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.633316 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z"] Nov 03 18:51:22.636485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.636463 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-32a59b\"" Nov 03 18:51:22.736865 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.736764 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2868fe39-a729-4b21-a27f-82c40f95de32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:22.737200 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.737175 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65k8\" (UniqueName: \"kubernetes.io/projected/2868fe39-a729-4b21-a27f-82c40f95de32-kube-api-access-v65k8\") pod \"payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:22.838012 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.837982 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2868fe39-a729-4b21-a27f-82c40f95de32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:22.838169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.838043 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65k8\" (UniqueName: \"kubernetes.io/projected/2868fe39-a729-4b21-a27f-82c40f95de32-kube-api-access-v65k8\") pod \"payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:22.838874 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.838854 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2868fe39-a729-4b21-a27f-82c40f95de32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:22.900966 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:22.900924 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65k8\" (UniqueName: \"kubernetes.io/projected/2868fe39-a729-4b21-a27f-82c40f95de32-kube-api-access-v65k8\") pod \"payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:23.195869 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:23.195833 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:23.584992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:23.584245 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z"] Nov 03 18:51:24.525517 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:24.525482 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" event={"ID":"2868fe39-a729-4b21-a27f-82c40f95de32","Type":"ContainerStarted","Data":"4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a"} Nov 03 18:51:24.525728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:24.525538 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" event={"ID":"2868fe39-a729-4b21-a27f-82c40f95de32","Type":"ContainerStarted","Data":"277bf53dd637884e6a413246e239bfe0cc595361b099748036c8b250461f114e"} Nov 03 18:51:24.525728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:24.525561 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:24.643638 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:24.643573 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" podStartSLOduration=2.643557434 podStartE2EDuration="2.643557434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:24.642760577 +0000 UTC m=+814.125756494" watchObservedRunningTime="2026-11-03 18:51:24.643557434 +0000 UTC m=+814.126553348" Nov 03 18:51:26.442959 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:26.442907 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 18:51:27.453162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.453135 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 18:51:27.704964 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.704885 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8"] Nov 03 18:51:27.707946 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.707925 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:27.761791 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.761760 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9686c3-dockercfg-4z4kc\"" Nov 03 18:51:27.761949 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.761851 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9686c3\"" Nov 03 18:51:27.765255 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.765222 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8"] Nov 03 18:51:27.862262 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.862212 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:27.862562 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.862541 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqqn\" (UniqueName: \"kubernetes.io/projected/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-kube-api-access-dvqqn\") pod \"payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:27.964298 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.964206 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:27.964446 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.964309 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqqn\" (UniqueName: \"kubernetes.io/projected/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-kube-api-access-dvqqn\") pod \"payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:27.965250 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:27.965220 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:28.028072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.028039 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqqn\" (UniqueName: \"kubernetes.io/projected/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-kube-api-access-dvqqn\") pod \"payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:28.321857 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.321810 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:28.553288 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.553203 4948 generic.go:356] "Generic (PLEG): container finished" podID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerID="70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90" exitCode=0 Nov 03 18:51:28.553288 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.553272 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" event={"ID":"4f639da3-66ce-4c7b-b94a-5f9edbf6651c","Type":"ContainerDied","Data":"70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90"} Nov 03 18:51:28.553801 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.553748 4948 scope.go:122] "RemoveContainer" containerID="70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90" Nov 03 18:51:28.555264 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.555236 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:28.555375 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.555303 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:28.555375 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.555333 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:28.742463 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:28.742442 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8"] Nov 03 18:51:29.486352 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:29.486323 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 18:51:29.486562 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:29.486403 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 18:51:29.581081 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:29.581046 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" event={"ID":"1fc2b876-e5c6-4008-9558-f3568dc1a6e8","Type":"ContainerStarted","Data":"5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238"} Nov 03 18:51:29.581477 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:29.581100 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" event={"ID":"1fc2b876-e5c6-4008-9558-f3568dc1a6e8","Type":"ContainerStarted","Data":"c738bcd0f58d985a69654b05e31c2a8ea7cbfc42bbf00a8e85005d295e3d0474"} Nov 03 18:51:29.581477 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:29.581122 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:29.583189 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:29.583158 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" event={"ID":"4f639da3-66ce-4c7b-b94a-5f9edbf6651c","Type":"ContainerStarted","Data":"7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630"} Nov 03 18:51:29.864071 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:29.863939 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" podStartSLOduration=2.863919375 podStartE2EDuration="2.863919375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:29.858800052 +0000 UTC m=+819.341795967" watchObservedRunningTime="2026-11-03 18:51:29.863919375 +0000 UTC m=+819.346915287" Nov 03 18:51:30.200729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:30.200676 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:30.201858 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:30.201829 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:51:30.202020 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:30.201881 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 18:51:31.200320 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:31.200281 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:51:31.200755 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:31.200351 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 18:51:31.303804 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:31.303766 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:32.207735 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:32.204074 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:51:32.207735 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:32.204141 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 18:51:32.507281 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:32.507194 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 18:51:32.919881 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:51:32.919772 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:51:33.023223 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.023186 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp"] Nov 03 18:51:33.028485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.028460 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.053729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.050064 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f42l2\" (UniqueName: \"kubernetes.io/projected/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-kube-api-access-f42l2\") pod \"payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.053729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.050137 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.098720 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.092108 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-3a7056\"" Nov 03 18:51:33.124733 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.123228 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-3a7056-dockercfg-w9bqx\"" Nov 03 18:51:33.125068 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.125040 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp"] Nov 03 18:51:33.176777 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.151120 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f42l2\" (UniqueName: \"kubernetes.io/projected/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-kube-api-access-f42l2\") pod \"payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.195889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.195836 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.197377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.197086 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.207243 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.206809 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:51:33.207243 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.206869 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 18:51:33.234692 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.234665 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f42l2\" (UniqueName: \"kubernetes.io/projected/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-kube-api-access-f42l2\") pod \"payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.363401 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.363110 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:33.825304 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:51:33.825260 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed68fe7_17e0_4678_bbe3_caf1e5d46ac2.slice/crio-a5d874760e02652e534a61848cc7bcdaf791f78a1dc253981a8ce27385209a28 WatchSource:0}: Error finding container a5d874760e02652e534a61848cc7bcdaf791f78a1dc253981a8ce27385209a28: Status 404 returned error can't find the container with id a5d874760e02652e534a61848cc7bcdaf791f78a1dc253981a8ce27385209a28 Nov 03 18:51:33.827056 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:33.827028 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp"] Nov 03 18:51:34.202746 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:34.201252 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:51:34.202746 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:34.201319 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 18:51:34.688725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:34.686725 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" event={"ID":"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2","Type":"ContainerStarted","Data":"7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d"} Nov 03 18:51:34.688725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:34.686800 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" event={"ID":"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2","Type":"ContainerStarted","Data":"a5d874760e02652e534a61848cc7bcdaf791f78a1dc253981a8ce27385209a28"} Nov 03 18:51:34.688725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:34.687781 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:34.811840 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:34.811778 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" podStartSLOduration=2.811761769 podStartE2EDuration="2.811761769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:34.808935634 +0000 UTC m=+824.291931554" watchObservedRunningTime="2026-11-03 18:51:34.811761769 +0000 UTC m=+824.294757683" Nov 03 18:51:35.205268 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:35.205237 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:35.206333 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:35.206310 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 18:51:35.535253 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:35.535182 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 18:51:40.592783 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:40.592752 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 18:51:41.745441 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:41.745403 4948 generic.go:356] "Generic (PLEG): container finished" podID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerID="490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc" exitCode=0 Nov 03 18:51:41.745890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:41.745471 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" event={"ID":"26d25ebd-3ae1-409d-9f84-70bb798ba6a8","Type":"ContainerDied","Data":"490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc"} Nov 03 18:51:41.745946 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:41.745903 4948 scope.go:122] "RemoveContainer" containerID="490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc" Nov 03 18:51:41.747233 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:41.747194 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:41.747359 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:41.747277 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:41.747434 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:41.747356 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:42.775729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:42.769851 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" event={"ID":"26d25ebd-3ae1-409d-9f84-70bb798ba6a8","Type":"ContainerStarted","Data":"5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530"} Nov 03 18:51:43.499836 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:43.499801 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:43.500024 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:43.500002 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:51:43.500068 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:43.500047 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:51:44.500152 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:44.500117 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:51:44.500775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:44.500740 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:51:45.504176 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:45.504134 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:51:45.504816 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:45.504778 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:51:46.504142 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:46.504102 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:51:46.504451 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:46.504425 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:51:46.709680 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:46.709649 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 18:51:47.500092 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:47.500054 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:51:47.500384 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:47.500357 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 18:51:48.313218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:48.313179 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:48.505316 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:48.505282 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:48.847538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:48.847505 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 18:51:53.904447 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:53.904408 4948 generic.go:356] "Generic (PLEG): container finished" podID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerID="2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691" exitCode=0 Nov 03 18:51:53.904951 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:53.904484 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" event={"ID":"dd255599-9154-4ee4-b1bf-3b44fb198f9a","Type":"ContainerDied","Data":"2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691"} Nov 03 18:51:53.904951 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:53.904912 4948 scope.go:122] "RemoveContainer" containerID="2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691" Nov 03 18:51:53.906338 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:53.906310 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:53.906420 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:53.906376 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:53.906420 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:53.906406 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:51:54.935040 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:54.935001 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" event={"ID":"dd255599-9154-4ee4-b1bf-3b44fb198f9a","Type":"ContainerStarted","Data":"a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40"} Nov 03 18:51:55.448243 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:55.448214 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:51:55.448407 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:55.448315 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 18:51:55.448407 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:55.448357 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 18:51:56.448871 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:56.448840 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 18:51:56.449323 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:56.449291 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 18:51:57.449071 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:57.449037 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 18:51:57.449470 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:57.449086 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 18:51:58.448496 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:58.448461 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 18:51:58.448681 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:58.448516 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 18:51:59.448936 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:59.448895 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 18:51:59.449328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:51:59.448964 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 18:52:00.452384 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:00.452353 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:52:00.452852 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:00.452811 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:52:00.453636 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:00.453613 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 18:52:02.344875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:02.344838 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:52:02.345271 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:02.344902 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:52:02.794213 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:52:02.794174 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:52:04.058263 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:04.058229 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerID="8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81" exitCode=0 Nov 03 18:52:04.058779 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:04.058303 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" event={"ID":"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a","Type":"ContainerDied","Data":"8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81"} Nov 03 18:52:04.058851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:04.058833 4948 scope.go:122] "RemoveContainer" containerID="8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81" Nov 03 18:52:04.060200 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:04.060170 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:52:04.060306 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:04.060252 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:52:04.060306 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:04.060299 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:52:05.081769 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:05.081691 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" event={"ID":"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a","Type":"ContainerStarted","Data":"6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8"} Nov 03 18:52:05.916940 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:05.916901 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:52:05.917134 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:05.917095 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:52:05.917182 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:05.917139 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 18:52:06.916879 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:06.916845 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:52:06.917286 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:06.916904 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 18:52:07.916897 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:07.916862 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:52:07.917301 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:07.916920 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 18:52:08.916465 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:08.916423 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:52:08.916663 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:08.916490 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 18:52:09.916407 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:09.916369 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:52:09.916939 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:09.916439 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 18:52:10.919844 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:10.919814 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:52:10.920299 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:10.920276 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:52:10.920998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:10.920975 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 18:52:32.793066 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:52:32.793028 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:52:50.846200 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:50.846170 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:52:50.848378 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:50.848356 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:52:50.853492 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:50.853471 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:52:50.855476 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:52:50.855459 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:53:02.793408 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:53:02.793367 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:53:30.585068 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.585033 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn"] Nov 03 18:53:30.588753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.588732 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.636443 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.636413 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90533a1d-b6cf-4600-886b-6429d04c9e2c-tls-certs\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.636604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.636467 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-tmp-dir\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.636604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.636544 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-home\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.636604 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.636573 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-model-cache\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.636794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.636621 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-dshm\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.636851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.636798 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vdg\" (UniqueName: \"kubernetes.io/projected/90533a1d-b6cf-4600-886b-6429d04c9e2c-kube-api-access-q9vdg\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.647731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.647340 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn"] Nov 03 18:53:30.694861 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.694772 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"simulator-w2-32a59b-kserve-self-signed-certs\"" Nov 03 18:53:30.694861 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.694802 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"default-dockercfg-zbhk5\"" Nov 03 18:53:30.694861 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.694792 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"openshift-service-ca.crt\"" Nov 03 18:53:30.694861 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.694838 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"kube-root-ca.crt\"" Nov 03 18:53:30.740729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.740300 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-dshm\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.740729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.740397 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vdg\" (UniqueName: \"kubernetes.io/projected/90533a1d-b6cf-4600-886b-6429d04c9e2c-kube-api-access-q9vdg\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.740729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.740469 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90533a1d-b6cf-4600-886b-6429d04c9e2c-tls-certs\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.740729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.740528 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-tmp-dir\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.740729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.740557 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-home\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.740729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.740581 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-model-cache\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.741242 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.741067 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-model-cache\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.754497 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.754456 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-tmp-dir\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.755037 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.755012 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-home\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.763022 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.761721 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-dshm\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.763914 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.763685 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90533a1d-b6cf-4600-886b-6429d04c9e2c-tls-certs\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.817259 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.817228 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vdg\" (UniqueName: \"kubernetes.io/projected/90533a1d-b6cf-4600-886b-6429d04c9e2c-kube-api-access-q9vdg\") pod \"simulator-w2-32a59b-kserve-6888947cb9-5k9rn\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:30.911850 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:30.911760 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:31.302202 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:53:31.302175 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90533a1d_b6cf_4600_886b_6429d04c9e2c.slice/crio-a99dd06029912fb69da421b99eec2da1fec463b790e947ebfb28701942553b6e WatchSource:0}: Error finding container a99dd06029912fb69da421b99eec2da1fec463b790e947ebfb28701942553b6e: Status 404 returned error can't find the container with id a99dd06029912fb69da421b99eec2da1fec463b790e947ebfb28701942553b6e Nov 03 18:53:31.302334 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:31.302247 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn"] Nov 03 18:53:31.659022 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:31.658926 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" event={"ID":"90533a1d-b6cf-4600-886b-6429d04c9e2c","Type":"ContainerStarted","Data":"67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81"} Nov 03 18:53:31.659022 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:31.658985 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" event={"ID":"90533a1d-b6cf-4600-886b-6429d04c9e2c","Type":"ContainerStarted","Data":"a99dd06029912fb69da421b99eec2da1fec463b790e947ebfb28701942553b6e"} Nov 03 18:53:31.780518 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:31.780466 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podStartSLOduration=1.78045375 podStartE2EDuration="1.78045375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:31.777618141 +0000 UTC m=+941.260614061" watchObservedRunningTime="2026-11-03 18:53:31.78045375 +0000 UTC m=+941.263449663" Nov 03 18:53:31.912228 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:31.912146 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:32.794324 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:53:32.794288 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:53:40.912617 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:40.912585 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:40.926061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:40.926037 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:41.736149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:41.736122 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 18:53:50.591690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.591656 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p"] Nov 03 18:53:50.595007 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.594989 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.679038 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.679007 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"kube-root-ca.crt\"" Nov 03 18:53:50.682483 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.682448 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def6a3c-a7b9-425c-8a92-c46cd6073694-tls-certs\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.682638 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.682506 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-tmp-dir\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.682734 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.682658 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xndgq\" (UniqueName: \"kubernetes.io/projected/4def6a3c-a7b9-425c-8a92-c46cd6073694-kube-api-access-xndgq\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.682809 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.682729 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-dshm\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.682809 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.682783 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-home\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.682913 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.682825 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-model-cache\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.692330 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.692298 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"openshift-service-ca.crt\"" Nov 03 18:53:50.699992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.699967 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"default-dockercfg-dbh7c\"" Nov 03 18:53:50.699992 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.699972 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"premium-w4-2e052a-kserve-self-signed-certs\"" Nov 03 18:53:50.703752 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.703729 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p"] Nov 03 18:53:50.783411 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783383 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-dshm\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.783411 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783413 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-home\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.783659 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783429 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-model-cache\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.783659 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783559 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def6a3c-a7b9-425c-8a92-c46cd6073694-tls-certs\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.783829 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783670 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-tmp-dir\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.783887 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783863 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-home\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.783887 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783877 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-model-cache\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.783989 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.783863 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xndgq\" (UniqueName: \"kubernetes.io/projected/4def6a3c-a7b9-425c-8a92-c46cd6073694-kube-api-access-xndgq\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.784244 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.784217 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-tmp-dir\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.786370 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.786345 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-dshm\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.786516 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.786493 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def6a3c-a7b9-425c-8a92-c46cd6073694-tls-certs\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.848376 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.848302 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xndgq\" (UniqueName: \"kubernetes.io/projected/4def6a3c-a7b9-425c-8a92-c46cd6073694-kube-api-access-xndgq\") pod \"premium-w4-2e052a-kserve-599744c99-94j4p\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:50.907018 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:50.906983 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:51.337960 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:51.337928 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p"] Nov 03 18:53:51.339641 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:53:51.339617 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4def6a3c_a7b9_425c_8a92_c46cd6073694.slice/crio-0a989af2fcb62387e5c015460c9d9049e871fbe00545654def616a060385c939 WatchSource:0}: Error finding container 0a989af2fcb62387e5c015460c9d9049e871fbe00545654def616a060385c939: Status 404 returned error can't find the container with id 0a989af2fcb62387e5c015460c9d9049e871fbe00545654def616a060385c939 Nov 03 18:53:51.798631 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:51.798592 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" event={"ID":"4def6a3c-a7b9-425c-8a92-c46cd6073694","Type":"ContainerStarted","Data":"ec20aad21efff7db79368e70ee73b30b76616a6ef021884574e705642a65af8e"} Nov 03 18:53:51.798631 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:51.798633 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" event={"ID":"4def6a3c-a7b9-425c-8a92-c46cd6073694","Type":"ContainerStarted","Data":"0a989af2fcb62387e5c015460c9d9049e871fbe00545654def616a060385c939"} Nov 03 18:53:51.907301 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:51.907273 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:53:51.918064 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:53:51.918015 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podStartSLOduration=1.918002285 podStartE2EDuration="1.918002285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:51.917688022 +0000 UTC m=+961.400683937" watchObservedRunningTime="2026-11-03 18:53:51.918002285 +0000 UTC m=+961.400998228" Nov 03 18:54:00.907916 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:00.907886 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:54:00.921093 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:00.921067 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:54:01.879552 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:01.879521 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 18:54:02.793424 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:54:02.793376 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:54:32.794037 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:54:32.793996 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:54:56.636811 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.636767 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4dg7z-trghh"] Nov 03 18:54:56.640481 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.640449 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.701807 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.701775 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4dg7z-trghh"] Nov 03 18:54:56.744388 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.744358 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4ngk\" (UniqueName: \"kubernetes.io/projected/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-kube-api-access-h4ngk\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.744545 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.744416 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-service-ca-bundle\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.744545 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.744522 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-trusted-ca-bundle\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.744621 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.744592 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-archives-path\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.747138 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.747115 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2q9jp\"" Nov 03 18:54:56.845550 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.845510 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-trusted-ca-bundle\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.845832 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.845795 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-archives-path\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.845958 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.845933 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4ngk\" (UniqueName: \"kubernetes.io/projected/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-kube-api-access-h4ngk\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.846049 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.846027 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-service-ca-bundle\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.846431 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.846402 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-archives-path\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.846760 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.846739 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-service-ca-bundle\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.846998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.846981 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-trusted-ca-bundle\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.909270 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.909204 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4ngk\" (UniqueName: \"kubernetes.io/projected/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-kube-api-access-h4ngk\") pod \"periodic-gathering-4dg7z-trghh\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:56.952599 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:56.952572 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:54:57.331373 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:57.331346 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4dg7z-trghh"] Nov 03 18:54:57.331499 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:54:57.331481 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc2d04_5ae0_4cf1_901b_30b1fac954ee.slice/crio-546a55d253a4c73d266284a4a87caaedeabe0c0f60aab5bcc442678d83ff91f7 WatchSource:0}: Error finding container 546a55d253a4c73d266284a4a87caaedeabe0c0f60aab5bcc442678d83ff91f7: Status 404 returned error can't find the container with id 546a55d253a4c73d266284a4a87caaedeabe0c0f60aab5bcc442678d83ff91f7 Nov 03 18:54:58.260623 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:58.260587 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dg7z-trghh" event={"ID":"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee","Type":"ContainerStarted","Data":"228fa206489bee09366bc71e9b2ef0a76ed60a389f322c160d16c4050a224ca2"} Nov 03 18:54:58.261070 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:58.260645 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dg7z-trghh" event={"ID":"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee","Type":"ContainerStarted","Data":"546a55d253a4c73d266284a4a87caaedeabe0c0f60aab5bcc442678d83ff91f7"} Nov 03 18:54:58.378164 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:54:58.378108 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4dg7z-trghh" podStartSLOduration=2.378094488 podStartE2EDuration="2.378094488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:58.377347596 +0000 UTC m=+1027.860343523" watchObservedRunningTime="2026-11-03 18:54:58.378094488 +0000 UTC m=+1027.861090437" Nov 03 18:55:02.795489 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:55:02.795400 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:55:10.389648 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.389616 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf"] Nov 03 18:55:10.392675 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.392657 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.448328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.448290 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf"] Nov 03 18:55:10.495575 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.495545 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:55:10.495851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.495832 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:55:10.495950 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.495833 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:55:10.496876 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.496854 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:55:10.500438 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.500415 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.500519 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.500480 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.500607 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.500591 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2sk8\" (UniqueName: \"kubernetes.io/projected/f0ac0551-c3c5-4275-825e-9618fefd5852-kube-api-access-j2sk8\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.602007 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.601974 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.602169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.602052 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.602169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.602105 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2sk8\" (UniqueName: \"kubernetes.io/projected/f0ac0551-c3c5-4275-825e-9618fefd5852-kube-api-access-j2sk8\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.602813 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.602747 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.604322 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.604300 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.664728 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.664636 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2sk8\" (UniqueName: \"kubernetes.io/projected/f0ac0551-c3c5-4275-825e-9618fefd5852-kube-api-access-j2sk8\") pod \"maas-api-revoke-sub-2deef9c0-l7qdf\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:10.706011 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:10.705985 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:11.086303 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.086265 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf"] Nov 03 18:55:11.086303 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:55:11.086286 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ac0551_c3c5_4275_825e_9618fefd5852.slice/crio-1dfc46e6e41ef81a9515e110a7150d525ad05354321ed25fa7108173b82c4489 WatchSource:0}: Error finding container 1dfc46e6e41ef81a9515e110a7150d525ad05354321ed25fa7108173b82c4489: Status 404 returned error can't find the container with id 1dfc46e6e41ef81a9515e110a7150d525ad05354321ed25fa7108173b82c4489 Nov 03 18:55:11.088727 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.088680 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:55:11.349749 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.349625 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" event={"ID":"f0ac0551-c3c5-4275-825e-9618fefd5852","Type":"ContainerStarted","Data":"1dfc46e6e41ef81a9515e110a7150d525ad05354321ed25fa7108173b82c4489"} Nov 03 18:55:11.678773 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.678655 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv"] Nov 03 18:55:11.684029 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.683999 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.739301 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.739266 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv"] Nov 03 18:55:11.785998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.785971 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"openshift-service-ca.crt\"" Nov 03 18:55:11.785998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.785969 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"kube-root-ca.crt\"" Nov 03 18:55:11.786219 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.786044 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"default-dockercfg-qbv72\"" Nov 03 18:55:11.788983 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.788966 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"premium-w6-fba266-kserve-self-signed-certs\"" Nov 03 18:55:11.817758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.815398 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-dshm\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.817758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.815468 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-model-cache\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.817758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.815682 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-home\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.817758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.815811 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-tmp-dir\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.817758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.815840 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv8cg\" (UniqueName: \"kubernetes.io/projected/c01e3f68-46a3-4088-8aab-31f0a9e86108-kube-api-access-jv8cg\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.817758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.815871 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01e3f68-46a3-4088-8aab-31f0a9e86108-tls-certs\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.916667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.916630 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-home\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.916875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.916693 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-tmp-dir\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.916875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.916733 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv8cg\" (UniqueName: \"kubernetes.io/projected/c01e3f68-46a3-4088-8aab-31f0a9e86108-kube-api-access-jv8cg\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.916875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.916761 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01e3f68-46a3-4088-8aab-31f0a9e86108-tls-certs\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.916875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.916854 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-dshm\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.917086 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.916897 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-model-cache\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.917141 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.917083 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-home\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.917196 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.917130 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-tmp-dir\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.917302 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.917283 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-model-cache\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.919053 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.919028 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-dshm\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.919266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.919247 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01e3f68-46a3-4088-8aab-31f0a9e86108-tls-certs\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.979959 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.979928 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv8cg\" (UniqueName: \"kubernetes.io/projected/c01e3f68-46a3-4088-8aab-31f0a9e86108-kube-api-access-jv8cg\") pod \"premium-w6-fba266-kserve-79bbcfb6d9-26cqv\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:11.995770 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:11.995680 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:12.409205 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:12.409176 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv"] Nov 03 18:55:12.409327 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:55:12.409180 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc01e3f68_46a3_4088_8aab_31f0a9e86108.slice/crio-049242e6ed48f4360dbf2cd94de78eb7a40f812f978c8d3697e1f0b56ec34cea WatchSource:0}: Error finding container 049242e6ed48f4360dbf2cd94de78eb7a40f812f978c8d3697e1f0b56ec34cea: Status 404 returned error can't find the container with id 049242e6ed48f4360dbf2cd94de78eb7a40f812f978c8d3697e1f0b56ec34cea Nov 03 18:55:13.370289 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:13.366626 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" event={"ID":"c01e3f68-46a3-4088-8aab-31f0a9e86108","Type":"ContainerStarted","Data":"9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1"} Nov 03 18:55:13.370289 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:13.366688 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" event={"ID":"c01e3f68-46a3-4088-8aab-31f0a9e86108","Type":"ContainerStarted","Data":"049242e6ed48f4360dbf2cd94de78eb7a40f812f978c8d3697e1f0b56ec34cea"} Nov 03 18:55:13.486915 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:13.486856 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podStartSLOduration=2.486836459 podStartE2EDuration="2.486836459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:13.48639383 +0000 UTC m=+1042.969389745" watchObservedRunningTime="2026-11-03 18:55:13.486836459 +0000 UTC m=+1042.969832376" Nov 03 18:55:13.996778 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:13.996745 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:18.416766 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:18.416666 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" event={"ID":"f0ac0551-c3c5-4275-825e-9618fefd5852","Type":"ContainerStarted","Data":"37a5686019a51b7d2a13f77dd3a2cce3f0a4af43726b8d23c0c62233252b8300"} Nov 03 18:55:18.540581 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:18.540505 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" podStartSLOduration=2.073139924 podStartE2EDuration="8.540486055s" totalImagesPullingTime="6.467346131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:18.535434541 +0000 UTC m=+1048.018430457" watchObservedRunningTime="2026-11-03 18:55:18.540486055 +0000 UTC m=+1048.023481969" Nov 03 18:55:19.425554 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:19.425516 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0ac0551-c3c5-4275-825e-9618fefd5852" containerID="37a5686019a51b7d2a13f77dd3a2cce3f0a4af43726b8d23c0c62233252b8300" exitCode=0 Nov 03 18:55:19.425977 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:19.425601 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" event={"ID":"f0ac0551-c3c5-4275-825e-9618fefd5852","Type":"ContainerDied","Data":"37a5686019a51b7d2a13f77dd3a2cce3f0a4af43726b8d23c0c62233252b8300"} Nov 03 18:55:20.585042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.585018 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:20.632523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.632488 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-openshift-ca\") pod \"f0ac0551-c3c5-4275-825e-9618fefd5852\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " Nov 03 18:55:20.632739 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.632550 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ac0551-c3c5-4275-825e-9618fefd5852-kube-api-access-j2sk8\" (UniqueName: \"kubernetes.io/projected/f0ac0551-c3c5-4275-825e-9618fefd5852-kube-api-access-j2sk8\") pod \"f0ac0551-c3c5-4275-825e-9618fefd5852\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " Nov 03 18:55:20.632739 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.632685 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-xks-ca\") pod \"f0ac0551-c3c5-4275-825e-9618fefd5852\" (UID: \"f0ac0551-c3c5-4275-825e-9618fefd5852\") " Nov 03 18:55:20.632944 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.632919 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-openshift-ca" pod "f0ac0551-c3c5-4275-825e-9618fefd5852" (UID: "f0ac0551-c3c5-4275-825e-9618fefd5852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:55:20.633160 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.633142 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:20.634838 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.634809 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-xks-ca" pod "f0ac0551-c3c5-4275-825e-9618fefd5852" (UID: "f0ac0551-c3c5-4275-825e-9618fefd5852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:55:20.635067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.635046 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ac0551-c3c5-4275-825e-9618fefd5852-kube-api-access-j2sk8" pod "f0ac0551-c3c5-4275-825e-9618fefd5852" (UID: "f0ac0551-c3c5-4275-825e-9618fefd5852"). InnerVolumeSpecName "kube-api-access-j2sk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:55:20.734237 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.734201 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2sk8\" (UniqueName: \"kubernetes.io/projected/f0ac0551-c3c5-4275-825e-9618fefd5852-kube-api-access-j2sk8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:20.734237 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:20.734234 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ac0551-c3c5-4275-825e-9618fefd5852-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:21.439874 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:21.439846 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" Nov 03 18:55:21.440053 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:21.439893 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf" event={"ID":"f0ac0551-c3c5-4275-825e-9618fefd5852","Type":"ContainerDied","Data":"1dfc46e6e41ef81a9515e110a7150d525ad05354321ed25fa7108173b82c4489"} Nov 03 18:55:21.440053 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:21.439939 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dfc46e6e41ef81a9515e110a7150d525ad05354321ed25fa7108173b82c4489" Nov 03 18:55:21.996664 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:21.996627 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:22.009580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:22.009553 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:22.458034 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:22.458002 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 18:55:27.791357 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:27.791325 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6"] Nov 03 18:55:27.791843 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:27.791557 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" containerID="cri-o://0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26" gracePeriod=30 Nov 03 18:55:32.113472 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.113429 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s"] Nov 03 18:55:32.114210 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.114189 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0ac0551-c3c5-4275-825e-9618fefd5852" containerName="revoke-keys" Nov 03 18:55:32.114263 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.114213 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ac0551-c3c5-4275-825e-9618fefd5852" containerName="revoke-keys" Nov 03 18:55:32.114392 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.114379 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ac0551-c3c5-4275-825e-9618fefd5852" containerName="revoke-keys" Nov 03 18:55:32.119446 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.119423 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.169977 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.169946 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s"] Nov 03 18:55:32.191556 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.191522 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.191745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.191610 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.191803 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.191768 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlpz\" (UniqueName: \"kubernetes.io/projected/c67a5a70-3f6d-4515-8504-0d908883fcdb-kube-api-access-lnlpz\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.217588 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.217557 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:55:32.217588 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.217559 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:55:32.217807 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.217571 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:55:32.224104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.224073 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:55:32.292766 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.292735 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.292918 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.292778 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.292918 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.292882 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlpz\" (UniqueName: \"kubernetes.io/projected/c67a5a70-3f6d-4515-8504-0d908883fcdb-kube-api-access-lnlpz\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.293490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.293464 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.295309 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.295287 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.355411 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.355384 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlpz\" (UniqueName: \"kubernetes.io/projected/c67a5a70-3f6d-4515-8504-0d908883fcdb-kube-api-access-lnlpz\") pod \"maas-api-revoke-sub-44b68e20-zf77s\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.434003 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.433976 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:32.794751 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:55:32.794641 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:55:32.817219 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:55:32.817171 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67a5a70_3f6d_4515_8504_0d908883fcdb.slice/crio-1e8654251e2904f08b5f66d66173f48e0501705e71490db5d28a3dee182901b3 WatchSource:0}: Error finding container 1e8654251e2904f08b5f66d66173f48e0501705e71490db5d28a3dee182901b3: Status 404 returned error can't find the container with id 1e8654251e2904f08b5f66d66173f48e0501705e71490db5d28a3dee182901b3 Nov 03 18:55:32.817404 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:32.817382 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s"] Nov 03 18:55:33.046454 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.046384 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:55:33.203556 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203524 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-envoy\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203662 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-certs\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203710 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-token\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-token\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203752 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-credential-socket\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203791 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-kube-api-access-h64ml\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-kube-api-access-h64ml\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203875 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-data\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203916 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-socket\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204014 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.203956 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istiod-ca-cert\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204387 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204018 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-podinfo\") pod \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\" (UID: \"5c7c3958-42f9-4047-a05a-4bed04b4b34b\") " Nov 03 18:55:33.204387 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204031 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-certs" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:55:33.204387 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204291 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-socket" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:55:33.204554 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204424 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istiod-ca-cert" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:55:33.204554 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204476 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.204554 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204497 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.204725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204556 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-credential-socket" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:55:33.204725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.204592 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-data" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:55:33.206186 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.206158 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-token" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:55:33.206283 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.206256 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-envoy" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:55:33.206453 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.206434 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-kube-api-access-h64ml" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "kube-api-access-h64ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:55:33.206664 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.206649 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-podinfo" pod "5c7c3958-42f9-4047-a05a-4bed04b4b34b" (UID: "5c7c3958-42f9-4047-a05a-4bed04b4b34b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:55:33.305280 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.305253 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.305280 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.305286 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.305424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.305303 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64ml\" (UniqueName: \"kubernetes.io/projected/5c7c3958-42f9-4047-a05a-4bed04b4b34b-kube-api-access-h64ml\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.305424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.305326 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.305424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.305340 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.305424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.305355 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.305424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.305369 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7c3958-42f9-4047-a05a-4bed04b4b34b-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:33.549899 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.549865 4948 generic.go:356] "Generic (PLEG): container finished" podID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerID="0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26" exitCode=0 Nov 03 18:55:33.550096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.549938 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" Nov 03 18:55:33.550096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.549941 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" event={"ID":"5c7c3958-42f9-4047-a05a-4bed04b4b34b","Type":"ContainerDied","Data":"0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26"} Nov 03 18:55:33.550096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.550089 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6" event={"ID":"5c7c3958-42f9-4047-a05a-4bed04b4b34b","Type":"ContainerDied","Data":"81cb371bcb83e93afa3f4ec507c15f90e81d0a1e7f675e1a47512abc61c0a19c"} Nov 03 18:55:33.550246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.550121 4948 scope.go:122] "RemoveContainer" containerID="0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26" Nov 03 18:55:33.552494 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.552297 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" event={"ID":"c67a5a70-3f6d-4515-8504-0d908883fcdb","Type":"ContainerStarted","Data":"fcd445ed247ae5937f83731e602d2cd5e6f8743079dcc40cb699155e5652584e"} Nov 03 18:55:33.552494 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.552348 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" event={"ID":"c67a5a70-3f6d-4515-8504-0d908883fcdb","Type":"ContainerStarted","Data":"1e8654251e2904f08b5f66d66173f48e0501705e71490db5d28a3dee182901b3"} Nov 03 18:55:33.561509 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.561488 4948 scope.go:122] "RemoveContainer" containerID="0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26" Nov 03 18:55:33.561827 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:55:33.561807 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26\": container with ID starting with 0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26 not found: ID does not exist" containerID="0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26" Nov 03 18:55:33.561881 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.561836 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26"} err="failed to get container status \"0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26\": rpc error: code = NotFound desc = could not find container \"0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26\": container with ID starting with 0bc50e084f39d213232bd8cb0b4bdd6723fb3fe5edf5a7c4a92cb152d892cc26 not found: ID does not exist" Nov 03 18:55:33.724060 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.724033 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6"] Nov 03 18:55:33.730399 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.730375 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-5rsd6"] Nov 03 18:55:33.846370 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:33.846274 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" podStartSLOduration=1.8462586920000001 podStartE2EDuration="1.846258692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:33.843959593 +0000 UTC m=+1063.326955508" watchObservedRunningTime="2026-11-03 18:55:33.846258692 +0000 UTC m=+1063.329254641" Nov 03 18:55:34.564507 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:34.564466 4948 generic.go:356] "Generic (PLEG): container finished" podID="c67a5a70-3f6d-4515-8504-0d908883fcdb" containerID="fcd445ed247ae5937f83731e602d2cd5e6f8743079dcc40cb699155e5652584e" exitCode=0 Nov 03 18:55:34.564987 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:34.564524 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" event={"ID":"c67a5a70-3f6d-4515-8504-0d908883fcdb","Type":"ContainerDied","Data":"fcd445ed247ae5937f83731e602d2cd5e6f8743079dcc40cb699155e5652584e"} Nov 03 18:55:34.754367 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:34.754333 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" path="/var/lib/kubelet/pods/5c7c3958-42f9-4047-a05a-4bed04b4b34b/volumes" Nov 03 18:55:35.727541 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.727507 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:35.847222 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.847131 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-xks-ca\") pod \"c67a5a70-3f6d-4515-8504-0d908883fcdb\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " Nov 03 18:55:35.847395 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.847289 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-openshift-ca\") pod \"c67a5a70-3f6d-4515-8504-0d908883fcdb\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " Nov 03 18:55:35.847395 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.847318 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67a5a70-3f6d-4515-8504-0d908883fcdb-kube-api-access-lnlpz\" (UniqueName: \"kubernetes.io/projected/c67a5a70-3f6d-4515-8504-0d908883fcdb-kube-api-access-lnlpz\") pod \"c67a5a70-3f6d-4515-8504-0d908883fcdb\" (UID: \"c67a5a70-3f6d-4515-8504-0d908883fcdb\") " Nov 03 18:55:35.847685 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.847652 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-openshift-ca" pod "c67a5a70-3f6d-4515-8504-0d908883fcdb" (UID: "c67a5a70-3f6d-4515-8504-0d908883fcdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:55:35.847925 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.847904 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:35.850072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.850049 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-xks-ca" pod "c67a5a70-3f6d-4515-8504-0d908883fcdb" (UID: "c67a5a70-3f6d-4515-8504-0d908883fcdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:55:35.850680 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.850658 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67a5a70-3f6d-4515-8504-0d908883fcdb-kube-api-access-lnlpz" pod "c67a5a70-3f6d-4515-8504-0d908883fcdb" (UID: "c67a5a70-3f6d-4515-8504-0d908883fcdb"). InnerVolumeSpecName "kube-api-access-lnlpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:55:35.948641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.948609 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnlpz\" (UniqueName: \"kubernetes.io/projected/c67a5a70-3f6d-4515-8504-0d908883fcdb-kube-api-access-lnlpz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:35.948641 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:35.948639 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67a5a70-3f6d-4515-8504-0d908883fcdb-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:36.263509 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.263468 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6"] Nov 03 18:55:36.264118 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.264099 4948 cpu_manager.go:414] "containerMap: removing container" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" Nov 03 18:55:36.264118 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.264117 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" Nov 03 18:55:36.264269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.264130 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c67a5a70-3f6d-4515-8504-0d908883fcdb" containerName="revoke-keys" Nov 03 18:55:36.264269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.264136 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67a5a70-3f6d-4515-8504-0d908883fcdb" containerName="revoke-keys" Nov 03 18:55:36.264269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.264266 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7c3958-42f9-4047-a05a-4bed04b4b34b" containerName="istio-proxy" Nov 03 18:55:36.264425 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.264276 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67a5a70-3f6d-4515-8504-0d908883fcdb" containerName="revoke-keys" Nov 03 18:55:36.267775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.267753 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.326036 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.325999 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6"] Nov 03 18:55:36.454675 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.454643 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.454873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.454779 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.454873 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.454835 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7rm\" (UniqueName: \"kubernetes.io/projected/a4f07be9-a414-4370-b5bc-3e738d78fcbd-kube-api-access-9q7rm\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.556226 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.556145 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7rm\" (UniqueName: \"kubernetes.io/projected/a4f07be9-a414-4370-b5bc-3e738d78fcbd-kube-api-access-9q7rm\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.556378 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.556316 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.556442 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.556383 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.557146 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.557119 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.558852 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.558832 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.580593 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.580570 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" Nov 03 18:55:36.580731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.580601 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s" event={"ID":"c67a5a70-3f6d-4515-8504-0d908883fcdb","Type":"ContainerDied","Data":"1e8654251e2904f08b5f66d66173f48e0501705e71490db5d28a3dee182901b3"} Nov 03 18:55:36.580731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.580654 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e8654251e2904f08b5f66d66173f48e0501705e71490db5d28a3dee182901b3" Nov 03 18:55:36.618918 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.618889 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7rm\" (UniqueName: \"kubernetes.io/projected/a4f07be9-a414-4370-b5bc-3e738d78fcbd-kube-api-access-9q7rm\") pod \"maas-api-revoke-sub-8be2364a-wmrx6\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:36.882453 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:36.882360 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:37.266918 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:37.266884 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6"] Nov 03 18:55:37.267464 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:55:37.267437 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f07be9_a414_4370_b5bc_3e738d78fcbd.slice/crio-1641cda8ce8e5df15340501c357895e3e5609262a0b56120482f88886dcddf7b WatchSource:0}: Error finding container 1641cda8ce8e5df15340501c357895e3e5609262a0b56120482f88886dcddf7b: Status 404 returned error can't find the container with id 1641cda8ce8e5df15340501c357895e3e5609262a0b56120482f88886dcddf7b Nov 03 18:55:37.593008 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:37.592976 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" event={"ID":"a4f07be9-a414-4370-b5bc-3e738d78fcbd","Type":"ContainerStarted","Data":"2c3859f58454c2ada2680cc4ebf8fb8e695d33f3693fc65c50dccbe0b1b413fd"} Nov 03 18:55:37.593191 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:37.593025 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" event={"ID":"a4f07be9-a414-4370-b5bc-3e738d78fcbd","Type":"ContainerStarted","Data":"1641cda8ce8e5df15340501c357895e3e5609262a0b56120482f88886dcddf7b"} Nov 03 18:55:37.714153 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:37.714102 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" podStartSLOduration=1.714088275 podStartE2EDuration="1.714088275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:37.711374503 +0000 UTC m=+1067.194370454" watchObservedRunningTime="2026-11-03 18:55:37.714088275 +0000 UTC m=+1067.197084217" Nov 03 18:55:38.607658 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:38.607578 4948 generic.go:356] "Generic (PLEG): container finished" podID="a4f07be9-a414-4370-b5bc-3e738d78fcbd" containerID="2c3859f58454c2ada2680cc4ebf8fb8e695d33f3693fc65c50dccbe0b1b413fd" exitCode=0 Nov 03 18:55:38.608027 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:38.607661 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" event={"ID":"a4f07be9-a414-4370-b5bc-3e738d78fcbd","Type":"ContainerDied","Data":"2c3859f58454c2ada2680cc4ebf8fb8e695d33f3693fc65c50dccbe0b1b413fd"} Nov 03 18:55:39.865853 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:39.865826 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:39.902888 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:39.902692 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f07be9-a414-4370-b5bc-3e738d78fcbd-kube-api-access-9q7rm\" (UniqueName: \"kubernetes.io/projected/a4f07be9-a414-4370-b5bc-3e738d78fcbd-kube-api-access-9q7rm\") pod \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " Nov 03 18:55:39.902888 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:39.902858 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-xks-ca\") pod \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " Nov 03 18:55:39.903120 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:39.903102 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-openshift-ca\") pod \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\" (UID: \"a4f07be9-a414-4370-b5bc-3e738d78fcbd\") " Nov 03 18:55:39.903824 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:39.903797 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-openshift-ca" pod "a4f07be9-a414-4370-b5bc-3e738d78fcbd" (UID: "a4f07be9-a414-4370-b5bc-3e738d78fcbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:55:39.905371 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:39.905345 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-xks-ca" pod "a4f07be9-a414-4370-b5bc-3e738d78fcbd" (UID: "a4f07be9-a414-4370-b5bc-3e738d78fcbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:55:39.905587 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:39.905559 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f07be9-a414-4370-b5bc-3e738d78fcbd-kube-api-access-9q7rm" pod "a4f07be9-a414-4370-b5bc-3e738d78fcbd" (UID: "a4f07be9-a414-4370-b5bc-3e738d78fcbd"). InnerVolumeSpecName "kube-api-access-9q7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:55:40.018753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:40.011721 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:40.018753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:40.011763 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f07be9-a414-4370-b5bc-3e738d78fcbd-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:40.018753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:40.011782 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7rm\" (UniqueName: \"kubernetes.io/projected/a4f07be9-a414-4370-b5bc-3e738d78fcbd-kube-api-access-9q7rm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:40.626935 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:40.626899 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" event={"ID":"a4f07be9-a414-4370-b5bc-3e738d78fcbd","Type":"ContainerDied","Data":"1641cda8ce8e5df15340501c357895e3e5609262a0b56120482f88886dcddf7b"} Nov 03 18:55:40.627121 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:40.626940 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6" Nov 03 18:55:40.627121 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:40.626948 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1641cda8ce8e5df15340501c357895e3e5609262a0b56120482f88886dcddf7b" Nov 03 18:55:47.590044 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.590011 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8"] Nov 03 18:55:47.590577 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.590563 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a4f07be9-a414-4370-b5bc-3e738d78fcbd" containerName="revoke-keys" Nov 03 18:55:47.590619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.590578 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f07be9-a414-4370-b5bc-3e738d78fcbd" containerName="revoke-keys" Nov 03 18:55:47.590682 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.590673 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f07be9-a414-4370-b5bc-3e738d78fcbd" containerName="revoke-keys" Nov 03 18:55:47.594589 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.594574 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.650880 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.650841 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8"] Nov 03 18:55:47.693539 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.693509 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d55p\" (UniqueName: \"kubernetes.io/projected/a777292e-b384-4354-9b75-54117e215b5f-kube-api-access-7d55p\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.693726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.693640 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777292e-b384-4354-9b75-54117e215b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.693780 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.693731 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777292e-b384-4354-9b75-54117e215b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.698295 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.698275 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:55:47.698498 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.698480 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:55:47.698556 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.698539 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:55:47.702650 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.702630 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:55:47.794262 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.794228 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777292e-b384-4354-9b75-54117e215b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.794262 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.794268 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d55p\" (UniqueName: \"kubernetes.io/projected/a777292e-b384-4354-9b75-54117e215b5f-kube-api-access-7d55p\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.794526 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.794343 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777292e-b384-4354-9b75-54117e215b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.795006 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.794982 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777292e-b384-4354-9b75-54117e215b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.796753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.796729 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777292e-b384-4354-9b75-54117e215b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.856693 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.856616 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d55p\" (UniqueName: \"kubernetes.io/projected/a777292e-b384-4354-9b75-54117e215b5f-kube-api-access-7d55p\") pod \"maas-api-revoke-sub-3f4f6a24-n5vr8\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:47.913179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:47.913146 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:48.296374 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:55:48.296344 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda777292e_b384_4354_9b75_54117e215b5f.slice/crio-2780f2053226da0874769f94f98c6bb49db3642c004efec1831bb10b00dd420b WatchSource:0}: Error finding container 2780f2053226da0874769f94f98c6bb49db3642c004efec1831bb10b00dd420b: Status 404 returned error can't find the container with id 2780f2053226da0874769f94f98c6bb49db3642c004efec1831bb10b00dd420b Nov 03 18:55:48.296681 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:48.296655 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8"] Nov 03 18:55:48.695510 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:48.695476 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" event={"ID":"a777292e-b384-4354-9b75-54117e215b5f","Type":"ContainerStarted","Data":"1aed2d476c7c58e28b3feed82ff6d6c62db1384bd1b56f395da8b717d39085cf"} Nov 03 18:55:48.695943 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:48.695524 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" event={"ID":"a777292e-b384-4354-9b75-54117e215b5f","Type":"ContainerStarted","Data":"2780f2053226da0874769f94f98c6bb49db3642c004efec1831bb10b00dd420b"} Nov 03 18:55:48.815976 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:48.815883 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" podStartSLOduration=1.8158702 podStartE2EDuration="1.8158702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:48.813539716 +0000 UTC m=+1078.296535632" watchObservedRunningTime="2026-11-03 18:55:48.8158702 +0000 UTC m=+1078.298866114" Nov 03 18:55:49.704374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:49.704331 4948 generic.go:356] "Generic (PLEG): container finished" podID="a777292e-b384-4354-9b75-54117e215b5f" containerID="1aed2d476c7c58e28b3feed82ff6d6c62db1384bd1b56f395da8b717d39085cf" exitCode=0 Nov 03 18:55:49.704803 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:49.704411 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" event={"ID":"a777292e-b384-4354-9b75-54117e215b5f","Type":"ContainerDied","Data":"1aed2d476c7c58e28b3feed82ff6d6c62db1384bd1b56f395da8b717d39085cf"} Nov 03 18:55:50.852235 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.852208 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:50.928083 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.928054 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a777292e-b384-4354-9b75-54117e215b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777292e-b384-4354-9b75-54117e215b5f-maas-api-xks-ca\") pod \"a777292e-b384-4354-9b75-54117e215b5f\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " Nov 03 18:55:50.928208 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.928110 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a777292e-b384-4354-9b75-54117e215b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777292e-b384-4354-9b75-54117e215b5f-maas-api-openshift-ca\") pod \"a777292e-b384-4354-9b75-54117e215b5f\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " Nov 03 18:55:50.928208 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.928158 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a777292e-b384-4354-9b75-54117e215b5f-kube-api-access-7d55p\" (UniqueName: \"kubernetes.io/projected/a777292e-b384-4354-9b75-54117e215b5f-kube-api-access-7d55p\") pod \"a777292e-b384-4354-9b75-54117e215b5f\" (UID: \"a777292e-b384-4354-9b75-54117e215b5f\") " Nov 03 18:55:50.928522 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.928501 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a777292e-b384-4354-9b75-54117e215b5f-maas-api-openshift-ca" pod "a777292e-b384-4354-9b75-54117e215b5f" (UID: "a777292e-b384-4354-9b75-54117e215b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:55:50.928646 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.928631 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a777292e-b384-4354-9b75-54117e215b5f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:50.930190 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.930158 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a777292e-b384-4354-9b75-54117e215b5f-maas-api-xks-ca" pod "a777292e-b384-4354-9b75-54117e215b5f" (UID: "a777292e-b384-4354-9b75-54117e215b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:55:50.930272 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:50.930255 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a777292e-b384-4354-9b75-54117e215b5f-kube-api-access-7d55p" pod "a777292e-b384-4354-9b75-54117e215b5f" (UID: "a777292e-b384-4354-9b75-54117e215b5f"). InnerVolumeSpecName "kube-api-access-7d55p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:55:51.029348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:51.029273 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a777292e-b384-4354-9b75-54117e215b5f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:51.029348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:51.029301 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d55p\" (UniqueName: \"kubernetes.io/projected/a777292e-b384-4354-9b75-54117e215b5f-kube-api-access-7d55p\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:55:51.719931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:51.719889 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" event={"ID":"a777292e-b384-4354-9b75-54117e215b5f","Type":"ContainerDied","Data":"2780f2053226da0874769f94f98c6bb49db3642c004efec1831bb10b00dd420b"} Nov 03 18:55:51.719931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:51.719909 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8" Nov 03 18:55:51.720178 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:51.719943 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2780f2053226da0874769f94f98c6bb49db3642c004efec1831bb10b00dd420b" Nov 03 18:55:56.409231 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.409185 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b"] Nov 03 18:55:56.409979 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.409957 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a777292e-b384-4354-9b75-54117e215b5f" containerName="revoke-keys" Nov 03 18:55:56.410091 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.409979 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a777292e-b384-4354-9b75-54117e215b5f" containerName="revoke-keys" Nov 03 18:55:56.410178 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.410164 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a777292e-b384-4354-9b75-54117e215b5f" containerName="revoke-keys" Nov 03 18:55:56.415484 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.415462 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.468256 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.468230 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b"] Nov 03 18:55:56.486597 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.486565 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjsq\" (UniqueName: \"kubernetes.io/projected/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-kube-api-access-pjjsq\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.486792 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.486636 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.486792 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.486742 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.515135 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.515110 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:55:56.515135 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.515110 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:55:56.515304 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.515142 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:55:56.519683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.519668 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:55:56.590783 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.590729 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjsq\" (UniqueName: \"kubernetes.io/projected/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-kube-api-access-pjjsq\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.590962 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.590867 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.590962 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.590914 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.592218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.592183 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.593887 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.593864 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.653667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.653631 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjsq\" (UniqueName: \"kubernetes.io/projected/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-kube-api-access-pjjsq\") pod \"maas-api-revoke-sub-ad344f19-tk42b\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:56.728914 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:56.728879 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:55:57.112412 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:57.112387 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b"] Nov 03 18:55:57.113113 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:55:57.113081 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ff7716_3db1_42bf_9bfa_b8f9e6ccf521.slice/crio-db4c2412988d1d3dddfdff4433d1f80c2c638bf61cf6e45203d4f7b822339512 WatchSource:0}: Error finding container db4c2412988d1d3dddfdff4433d1f80c2c638bf61cf6e45203d4f7b822339512: Status 404 returned error can't find the container with id db4c2412988d1d3dddfdff4433d1f80c2c638bf61cf6e45203d4f7b822339512 Nov 03 18:55:57.765477 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:57.765435 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" event={"ID":"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521","Type":"ContainerStarted","Data":"28098d48ce27dc6827b2cc23816548ae334207ac9dd36170b7b973dee204ae6b"} Nov 03 18:55:57.765932 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:57.765497 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" event={"ID":"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521","Type":"ContainerStarted","Data":"db4c2412988d1d3dddfdff4433d1f80c2c638bf61cf6e45203d4f7b822339512"} Nov 03 18:55:57.884711 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:57.884644 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" podStartSLOduration=1.88463103 podStartE2EDuration="1.88463103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:57.883122373 +0000 UTC m=+1087.366118291" watchObservedRunningTime="2026-11-03 18:55:57.88463103 +0000 UTC m=+1087.367626989" Nov 03 18:55:58.776317 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:58.776279 4948 generic.go:356] "Generic (PLEG): container finished" podID="84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" containerID="28098d48ce27dc6827b2cc23816548ae334207ac9dd36170b7b973dee204ae6b" exitCode=0 Nov 03 18:55:58.776784 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:58.776366 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" event={"ID":"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521","Type":"ContainerDied","Data":"28098d48ce27dc6827b2cc23816548ae334207ac9dd36170b7b973dee204ae6b"} Nov 03 18:55:59.928087 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:55:59.928058 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:56:00.029422 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.029387 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-kube-api-access-pjjsq\" (UniqueName: \"kubernetes.io/projected/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-kube-api-access-pjjsq\") pod \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " Nov 03 18:56:00.029606 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.029492 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-openshift-ca\") pod \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " Nov 03 18:56:00.029606 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.029571 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-xks-ca\") pod \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\" (UID: \"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521\") " Nov 03 18:56:00.029908 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.029881 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-openshift-ca" pod "84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" (UID: "84ff7716-3db1-42bf-9bfa-b8f9e6ccf521"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:00.029996 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.029980 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:00.031574 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.031550 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-xks-ca" pod "84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" (UID: "84ff7716-3db1-42bf-9bfa-b8f9e6ccf521"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:00.031672 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.031645 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-kube-api-access-pjjsq" pod "84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" (UID: "84ff7716-3db1-42bf-9bfa-b8f9e6ccf521"). InnerVolumeSpecName "kube-api-access-pjjsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:00.131260 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.131228 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjjsq\" (UniqueName: \"kubernetes.io/projected/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-kube-api-access-pjjsq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:00.131260 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.131257 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:00.791166 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.791137 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" Nov 03 18:56:00.791316 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.791165 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b" event={"ID":"84ff7716-3db1-42bf-9bfa-b8f9e6ccf521","Type":"ContainerDied","Data":"db4c2412988d1d3dddfdff4433d1f80c2c638bf61cf6e45203d4f7b822339512"} Nov 03 18:56:00.791316 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:00.791215 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db4c2412988d1d3dddfdff4433d1f80c2c638bf61cf6e45203d4f7b822339512" Nov 03 18:56:01.769194 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.769151 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh"] Nov 03 18:56:01.770063 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.770036 4948 cpu_manager.go:414] "containerMap: removing container" podUID="84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" containerName="revoke-keys" Nov 03 18:56:01.770063 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.770062 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" containerName="revoke-keys" Nov 03 18:56:01.770269 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.770251 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" containerName="revoke-keys" Nov 03 18:56:01.774637 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.774612 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.841689 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.841654 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a4cacd-openshift-default-dockercfg-hkxvc\"" Nov 03 18:56:01.845867 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.845835 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh"] Nov 03 18:56:01.855015 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.854988 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855168 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855146 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855221 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855184 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855221 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855206 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855300 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855222 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855362 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mp7\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-kube-api-access-m5mp7\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855440 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855426 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855474 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855463 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.855505 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.855484 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957135 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957103 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957341 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957143 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957341 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957168 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957341 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957192 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957341 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957220 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mp7\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-kube-api-access-m5mp7\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957341 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957239 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957341 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957280 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957341 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957302 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957361 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957589 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957731 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957621 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957849 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957818 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.957893 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.957878 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.958137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.958095 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.959670 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.959649 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:01.960029 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:01.960009 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:02.019788 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.019719 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:02.019920 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.019799 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mp7\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-kube-api-access-m5mp7\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:02.090335 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.090301 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:02.525479 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.525449 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh"] Nov 03 18:56:02.526902 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:02.526876 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bed544_d72a_48d8_94fc_9c4f9707d16f.slice/crio-7bc611c9a6d9e24b6bdad16960d33a9be143226e8d57cae7d42a65eed4f7e638 WatchSource:0}: Error finding container 7bc611c9a6d9e24b6bdad16960d33a9be143226e8d57cae7d42a65eed4f7e638: Status 404 returned error can't find the container with id 7bc611c9a6d9e24b6bdad16960d33a9be143226e8d57cae7d42a65eed4f7e638 Nov 03 18:56:02.529246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.529218 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:56:02.529327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.529280 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:56:02.529327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.529311 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:56:02.795717 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:56:02.795607 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:56:02.809953 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.809917 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" event={"ID":"d3bed544-d72a-48d8-94fc-9c4f9707d16f","Type":"ContainerStarted","Data":"bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2"} Nov 03 18:56:02.810145 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.809974 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" event={"ID":"d3bed544-d72a-48d8-94fc-9c4f9707d16f","Type":"ContainerStarted","Data":"7bc611c9a6d9e24b6bdad16960d33a9be143226e8d57cae7d42a65eed4f7e638"} Nov 03 18:56:02.947822 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.947784 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst"] Nov 03 18:56:02.952244 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.952222 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.968757 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.968727 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.968880 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.968771 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.968880 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.968802 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.968995 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.968918 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.969060 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.969002 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.969060 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.969039 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.969179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.969075 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.969179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.969095 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.969179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.969170 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-kube-api-access-98wlv\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:02.998741 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:02.998666 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podStartSLOduration=1.998645456 podStartE2EDuration="1.998645456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:02.946084282 +0000 UTC m=+1092.429080229" watchObservedRunningTime="2026-11-03 18:56:02.998645456 +0000 UTC m=+1092.481641384" Nov 03 18:56:03.007327 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.007285 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3a7056-openshift-default-dockercfg-wjcwp\"" Nov 03 18:56:03.044169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.044134 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst"] Nov 03 18:56:03.071449 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.071350 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.075747 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.071680 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.076029 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.076005 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.076169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.076149 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.080925 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.078257 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.081385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.081072 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.081385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.081100 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.081385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.081166 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.081385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.081207 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.081385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.081242 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.081385 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.081300 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-kube-api-access-98wlv\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.081934 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.081913 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.082162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.082145 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.082408 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.082388 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.082622 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.082606 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.084335 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.084299 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.091071 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.091045 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:03.092404 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.092379 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:56:03.092501 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.092445 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:56:03.147440 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.147401 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.147588 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.147459 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-kube-api-access-98wlv\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.279122 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.279078 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:03.714764 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.714732 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst"] Nov 03 18:56:03.714972 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:03.714950 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb68ae87_0d06_47ee_bd24_c2b291d0ff8f.slice/crio-7371717cb4990bec77cda007b36924fa41df57c08f0267a9c8d16857ea8fb3da WatchSource:0}: Error finding container 7371717cb4990bec77cda007b36924fa41df57c08f0267a9c8d16857ea8fb3da: Status 404 returned error can't find the container with id 7371717cb4990bec77cda007b36924fa41df57c08f0267a9c8d16857ea8fb3da Nov 03 18:56:03.717352 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.717318 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:56:03.717457 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.717392 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:56:03.717457 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.717434 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:56:03.821036 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.820988 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" event={"ID":"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f","Type":"ContainerStarted","Data":"011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8"} Nov 03 18:56:03.821424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.821069 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" event={"ID":"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f","Type":"ContainerStarted","Data":"7371717cb4990bec77cda007b36924fa41df57c08f0267a9c8d16857ea8fb3da"} Nov 03 18:56:03.948690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:03.948638 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podStartSLOduration=1.948623978 podStartE2EDuration="1.948623978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:03.945530523 +0000 UTC m=+1093.428526460" watchObservedRunningTime="2026-11-03 18:56:03.948623978 +0000 UTC m=+1093.431619898" Nov 03 18:56:04.091131 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:04.091047 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:56:04.091131 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:04.091105 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:56:04.281945 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:04.281905 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:04.283863 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:04.283827 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:56:04.284142 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:04.284112 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 18:56:05.091001 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:05.090964 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:56:05.091995 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:05.091565 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:56:05.283732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:05.283106 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:56:05.283732 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:05.283166 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 18:56:06.109896 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:06.098811 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:56:06.109896 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:06.098873 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:56:06.283140 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:06.283102 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:56:06.283430 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:06.283399 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 18:56:07.093149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.093115 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:56:07.093439 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.093410 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:56:07.235148 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.235116 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv"] Nov 03 18:56:07.243158 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.243139 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.281711 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.281663 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:56:07.282019 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.281985 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 18:56:07.318173 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.318134 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv"] Nov 03 18:56:07.336554 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.336487 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zvz\" (UniqueName: \"kubernetes.io/projected/2980a7cb-b1fa-462d-b22c-40888df69c99-kube-api-access-x4zvz\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.336994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.336967 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.337185 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.337159 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.345870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.345795 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:56:07.346115 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.346092 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:56:07.346249 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.346235 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:56:07.346353 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.346330 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:56:07.437979 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.437941 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.438154 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.438079 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.438154 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.438134 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zvz\" (UniqueName: \"kubernetes.io/projected/2980a7cb-b1fa-462d-b22c-40888df69c99-kube-api-access-x4zvz\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.438765 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.438740 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.440866 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.440838 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.507375 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.507334 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zvz\" (UniqueName: \"kubernetes.io/projected/2980a7cb-b1fa-462d-b22c-40888df69c99-kube-api-access-x4zvz\") pod \"maas-api-revoke-sub-fbead74c-w66gv\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:07.622451 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:07.622358 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:08.095722 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:08.077503 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv"] Nov 03 18:56:08.096043 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:08.092219 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:56:08.096219 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:08.096191 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:56:08.097213 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:08.097181 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2980a7cb_b1fa_462d_b22c_40888df69c99.slice/crio-ce92bf0027d321eb66b7a758b69284fa614ceb3077a8c1d3ecf2c3093ed88b46 WatchSource:0}: Error finding container ce92bf0027d321eb66b7a758b69284fa614ceb3077a8c1d3ecf2c3093ed88b46: Status 404 returned error can't find the container with id ce92bf0027d321eb66b7a758b69284fa614ceb3077a8c1d3ecf2c3093ed88b46 Nov 03 18:56:08.283538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:08.283072 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:56:08.283538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:08.283146 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 18:56:08.919308 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:08.919265 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" event={"ID":"2980a7cb-b1fa-462d-b22c-40888df69c99","Type":"ContainerStarted","Data":"a7101d4c8ab3fb68220c78ad87c35fe0fb72151a247844df924b040e97d29123"} Nov 03 18:56:08.919514 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:08.919320 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" event={"ID":"2980a7cb-b1fa-462d-b22c-40888df69c99","Type":"ContainerStarted","Data":"ce92bf0027d321eb66b7a758b69284fa614ceb3077a8c1d3ecf2c3093ed88b46"} Nov 03 18:56:09.040220 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:09.040160 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" podStartSLOduration=2.040146235 podStartE2EDuration="2.040146235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:09.038470062 +0000 UTC m=+1098.521465979" watchObservedRunningTime="2026-11-03 18:56:09.040146235 +0000 UTC m=+1098.523142149" Nov 03 18:56:09.090829 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:09.090786 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:56:09.091030 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:09.090835 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:56:09.282281 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:09.280114 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:56:09.282281 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:09.280175 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 18:56:09.930128 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:09.930090 4948 generic.go:356] "Generic (PLEG): container finished" podID="2980a7cb-b1fa-462d-b22c-40888df69c99" containerID="a7101d4c8ab3fb68220c78ad87c35fe0fb72151a247844df924b040e97d29123" exitCode=0 Nov 03 18:56:09.930610 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:09.930234 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" event={"ID":"2980a7cb-b1fa-462d-b22c-40888df69c99","Type":"ContainerDied","Data":"a7101d4c8ab3fb68220c78ad87c35fe0fb72151a247844df924b040e97d29123"} Nov 03 18:56:10.096055 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:10.096024 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:10.096405 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:10.096379 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:10.097538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:10.097518 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 18:56:10.557916 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:10.557886 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:10.559718 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:10.559597 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:10.560785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:10.560763 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 18:56:11.089751 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.089721 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:11.219999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.218977 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2980a7cb-b1fa-462d-b22c-40888df69c99-kube-api-access-x4zvz\" (UniqueName: \"kubernetes.io/projected/2980a7cb-b1fa-462d-b22c-40888df69c99-kube-api-access-x4zvz\") pod \"2980a7cb-b1fa-462d-b22c-40888df69c99\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " Nov 03 18:56:11.220794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.220759 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-openshift-ca\") pod \"2980a7cb-b1fa-462d-b22c-40888df69c99\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " Nov 03 18:56:11.221087 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.221061 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-xks-ca\") pod \"2980a7cb-b1fa-462d-b22c-40888df69c99\" (UID: \"2980a7cb-b1fa-462d-b22c-40888df69c99\") " Nov 03 18:56:11.221305 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.221275 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-openshift-ca" pod "2980a7cb-b1fa-462d-b22c-40888df69c99" (UID: "2980a7cb-b1fa-462d-b22c-40888df69c99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:11.221573 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.221550 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:11.221892 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.221865 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2980a7cb-b1fa-462d-b22c-40888df69c99-kube-api-access-x4zvz" pod "2980a7cb-b1fa-462d-b22c-40888df69c99" (UID: "2980a7cb-b1fa-462d-b22c-40888df69c99"). InnerVolumeSpecName "kube-api-access-x4zvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:11.223889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.223858 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-xks-ca" pod "2980a7cb-b1fa-462d-b22c-40888df69c99" (UID: "2980a7cb-b1fa-462d-b22c-40888df69c99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:11.323238 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.323207 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2980a7cb-b1fa-462d-b22c-40888df69c99-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:11.323238 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.323237 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4zvz\" (UniqueName: \"kubernetes.io/projected/2980a7cb-b1fa-462d-b22c-40888df69c99-kube-api-access-x4zvz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:11.966254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.966217 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" Nov 03 18:56:11.966448 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.966270 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv" event={"ID":"2980a7cb-b1fa-462d-b22c-40888df69c99","Type":"ContainerDied","Data":"ce92bf0027d321eb66b7a758b69284fa614ceb3077a8c1d3ecf2c3093ed88b46"} Nov 03 18:56:11.966448 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:11.966359 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce92bf0027d321eb66b7a758b69284fa614ceb3077a8c1d3ecf2c3093ed88b46" Nov 03 18:56:27.445615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.445578 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g"] Nov 03 18:56:27.446377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.446162 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2980a7cb-b1fa-462d-b22c-40888df69c99" containerName="revoke-keys" Nov 03 18:56:27.446377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.446174 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2980a7cb-b1fa-462d-b22c-40888df69c99" containerName="revoke-keys" Nov 03 18:56:27.446377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.446255 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2980a7cb-b1fa-462d-b22c-40888df69c99" containerName="revoke-keys" Nov 03 18:56:27.451066 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.451041 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.504266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.504231 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g"] Nov 03 18:56:27.505044 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.505011 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.505174 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.505155 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.505218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.505197 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjgb\" (UniqueName: \"kubernetes.io/projected/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-kube-api-access-vkjgb\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.551729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.551669 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:56:27.551729 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.551679 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:56:27.551931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.551740 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:56:27.555994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.555976 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:56:27.605870 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.605839 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.606030 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.605885 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjgb\" (UniqueName: \"kubernetes.io/projected/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-kube-api-access-vkjgb\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.606102 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.606033 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.606633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.606607 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.608300 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.608277 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.667979 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.667945 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjgb\" (UniqueName: \"kubernetes.io/projected/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-kube-api-access-vkjgb\") pod \"maas-api-revoke-sub-5f72f05d-tx59g\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:27.767886 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:27.767814 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:28.146017 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:28.145975 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f59cf4_27b4_4c6d_93bc_62682bf0a3f6.slice/crio-5381f07d4316f9d5fc947135aa2644e7421407523574d5c77698ac37192a950e WatchSource:0}: Error finding container 5381f07d4316f9d5fc947135aa2644e7421407523574d5c77698ac37192a950e: Status 404 returned error can't find the container with id 5381f07d4316f9d5fc947135aa2644e7421407523574d5c77698ac37192a950e Nov 03 18:56:28.149136 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:28.149108 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g"] Nov 03 18:56:29.092880 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:29.092852 4948 generic.go:356] "Generic (PLEG): container finished" podID="81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" containerID="fe527ef832b50c33ab23d6c15b16d2de5b3c46caba928224d2959859bfecf378" exitCode=0 Nov 03 18:56:29.093257 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:29.092941 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" event={"ID":"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6","Type":"ContainerDied","Data":"fe527ef832b50c33ab23d6c15b16d2de5b3c46caba928224d2959859bfecf378"} Nov 03 18:56:29.093257 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:29.093045 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" event={"ID":"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6","Type":"ContainerStarted","Data":"5381f07d4316f9d5fc947135aa2644e7421407523574d5c77698ac37192a950e"} Nov 03 18:56:30.275217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.275191 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:30.341415 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.341380 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-kube-api-access-vkjgb\" (UniqueName: \"kubernetes.io/projected/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-kube-api-access-vkjgb\") pod \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " Nov 03 18:56:30.341607 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.341449 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-xks-ca\") pod \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " Nov 03 18:56:30.341607 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.341484 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-openshift-ca\") pod \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\" (UID: \"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6\") " Nov 03 18:56:30.341930 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.341900 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-openshift-ca" pod "81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" (UID: "81f59cf4-27b4-4c6d-93bc-62682bf0a3f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:30.343800 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.343681 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-xks-ca" pod "81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" (UID: "81f59cf4-27b4-4c6d-93bc-62682bf0a3f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:30.343800 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.343692 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-kube-api-access-vkjgb" pod "81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" (UID: "81f59cf4-27b4-4c6d-93bc-62682bf0a3f6"). InnerVolumeSpecName "kube-api-access-vkjgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:30.443420 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.443388 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjgb\" (UniqueName: \"kubernetes.io/projected/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-kube-api-access-vkjgb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:30.443420 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.443421 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:30.443616 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:30.443438 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:31.113377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:31.113290 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" event={"ID":"81f59cf4-27b4-4c6d-93bc-62682bf0a3f6","Type":"ContainerDied","Data":"5381f07d4316f9d5fc947135aa2644e7421407523574d5c77698ac37192a950e"} Nov 03 18:56:31.113377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:31.113311 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g" Nov 03 18:56:31.113377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:31.113342 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5381f07d4316f9d5fc947135aa2644e7421407523574d5c77698ac37192a950e" Nov 03 18:56:32.793755 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:56:32.793691 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:56:33.556448 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.556407 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk"] Nov 03 18:56:33.557400 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.557193 4948 cpu_manager.go:414] "containerMap: removing container" podUID="81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" containerName="revoke-keys" Nov 03 18:56:33.557400 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.557212 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" containerName="revoke-keys" Nov 03 18:56:33.557400 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.557312 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" containerName="revoke-keys" Nov 03 18:56:33.561345 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.561290 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.614388 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.614352 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk"] Nov 03 18:56:33.615424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.615383 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:56:33.661646 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.661609 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:56:33.661937 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.661917 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:56:33.666580 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.666564 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:56:33.679079 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.679056 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmp7\" (UniqueName: \"kubernetes.io/projected/04889421-f5e1-4a7f-b19d-2da6488c4b8e-kube-api-access-6gmp7\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.679192 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.679099 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.679271 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.679254 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.780819 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.780786 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmp7\" (UniqueName: \"kubernetes.io/projected/04889421-f5e1-4a7f-b19d-2da6488c4b8e-kube-api-access-6gmp7\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.781024 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.780833 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.781024 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.780921 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.781672 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.781647 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.783301 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.783282 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.843660 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.843588 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmp7\" (UniqueName: \"kubernetes.io/projected/04889421-f5e1-4a7f-b19d-2da6488c4b8e-kube-api-access-6gmp7\") pod \"maas-api-revoke-sub-2709ff2b-dmxhk\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:33.934775 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:33.934730 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:34.135538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.135452 4948 generic.go:356] "Generic (PLEG): container finished" podID="bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" containerID="228fa206489bee09366bc71e9b2ef0a76ed60a389f322c160d16c4050a224ca2" exitCode=0 Nov 03 18:56:34.135538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.135524 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dg7z-trghh" event={"ID":"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee","Type":"ContainerDied","Data":"228fa206489bee09366bc71e9b2ef0a76ed60a389f322c160d16c4050a224ca2"} Nov 03 18:56:34.185898 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.185869 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v"] Nov 03 18:56:34.191113 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.191094 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.252391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.252348 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v"] Nov 03 18:56:34.287063 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.287026 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.287063 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.287073 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkd7n\" (UniqueName: \"kubernetes.io/projected/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-kube-api-access-kkd7n\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.287305 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.287275 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.363650 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.363625 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk"] Nov 03 18:56:34.364514 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:34.364484 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04889421_f5e1_4a7f_b19d_2da6488c4b8e.slice/crio-d1b8f5b14a673b0a6fb1c6633a764078073b91e7b630e51264b1f16e8f69f83f WatchSource:0}: Error finding container d1b8f5b14a673b0a6fb1c6633a764078073b91e7b630e51264b1f16e8f69f83f: Status 404 returned error can't find the container with id d1b8f5b14a673b0a6fb1c6633a764078073b91e7b630e51264b1f16e8f69f83f Nov 03 18:56:34.388069 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.388015 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.388069 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.388046 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkd7n\" (UniqueName: \"kubernetes.io/projected/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-kube-api-access-kkd7n\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.388241 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.388226 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.388937 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.388917 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.390255 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.390235 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.450254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.450228 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkd7n\" (UniqueName: \"kubernetes.io/projected/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-kube-api-access-kkd7n\") pod \"maas-api-revoke-sub-2c738e92-rmn9v\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.505689 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.505662 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:34.889219 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:34.889189 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v"] Nov 03 18:56:34.889736 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:34.889687 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df5c7ca_9b8f_43e8_b283_62238b64cd6b.slice/crio-236319717c291b96015e17a890031af49a5964ec522e6946fa4ebc6f58af7a9a WatchSource:0}: Error finding container 236319717c291b96015e17a890031af49a5964ec522e6946fa4ebc6f58af7a9a: Status 404 returned error can't find the container with id 236319717c291b96015e17a890031af49a5964ec522e6946fa4ebc6f58af7a9a Nov 03 18:56:35.143684 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.143652 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" event={"ID":"3df5c7ca-9b8f-43e8-b283-62238b64cd6b","Type":"ContainerStarted","Data":"236319717c291b96015e17a890031af49a5964ec522e6946fa4ebc6f58af7a9a"} Nov 03 18:56:35.145357 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.145326 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" event={"ID":"04889421-f5e1-4a7f-b19d-2da6488c4b8e","Type":"ContainerStarted","Data":"e3f95ea6a12159834d048fb89372e58e9729032fe4c214019d6bd6700101dacd"} Nov 03 18:56:35.145488 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.145369 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" event={"ID":"04889421-f5e1-4a7f-b19d-2da6488c4b8e","Type":"ContainerStarted","Data":"d1b8f5b14a673b0a6fb1c6633a764078073b91e7b630e51264b1f16e8f69f83f"} Nov 03 18:56:35.267463 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.267415 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" podStartSLOduration=2.26740206 podStartE2EDuration="2.26740206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:35.263275646 +0000 UTC m=+1124.746271560" watchObservedRunningTime="2026-11-03 18:56:35.26740206 +0000 UTC m=+1124.750397973" Nov 03 18:56:35.319097 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.319069 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:56:35.412919 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.412889 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-service-ca-bundle\") pod \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " Nov 03 18:56:35.413077 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.412966 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-trusted-ca-bundle\") pod \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " Nov 03 18:56:35.413077 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413041 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-kube-api-access-h4ngk\" (UniqueName: \"kubernetes.io/projected/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-kube-api-access-h4ngk\") pod \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " Nov 03 18:56:35.413077 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413066 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-archives-path\") pod \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\" (UID: \"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee\") " Nov 03 18:56:35.413266 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413235 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-service-ca-bundle" pod "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" (UID: "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:35.413409 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413385 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-trusted-ca-bundle" pod "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" (UID: "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:35.413479 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413431 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-archives-path" pod "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" (UID: "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:56:35.413542 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413532 4948 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-archives-path\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:35.413595 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413547 4948 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-service-ca-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:35.413595 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.413556 4948 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-trusted-ca-bundle\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:35.415448 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.415422 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-kube-api-access-h4ngk" pod "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" (UID: "bcfc2d04-5ae0-4cf1-901b-30b1fac954ee"). InnerVolumeSpecName "kube-api-access-h4ngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:35.514710 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:35.514619 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4ngk\" (UniqueName: \"kubernetes.io/projected/bcfc2d04-5ae0-4cf1-901b-30b1fac954ee-kube-api-access-h4ngk\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:36.153663 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:36.153566 4948 generic.go:356] "Generic (PLEG): container finished" podID="3df5c7ca-9b8f-43e8-b283-62238b64cd6b" containerID="cf0ab181fae2c37b8cbaefb845c8727850ecce81bcc204a42800a89abfcb7da9" exitCode=0 Nov 03 18:56:36.154072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:36.153655 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" event={"ID":"3df5c7ca-9b8f-43e8-b283-62238b64cd6b","Type":"ContainerDied","Data":"cf0ab181fae2c37b8cbaefb845c8727850ecce81bcc204a42800a89abfcb7da9"} Nov 03 18:56:36.155415 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:36.155391 4948 generic.go:356] "Generic (PLEG): container finished" podID="04889421-f5e1-4a7f-b19d-2da6488c4b8e" containerID="e3f95ea6a12159834d048fb89372e58e9729032fe4c214019d6bd6700101dacd" exitCode=0 Nov 03 18:56:36.155622 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:36.155479 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" event={"ID":"04889421-f5e1-4a7f-b19d-2da6488c4b8e","Type":"ContainerDied","Data":"e3f95ea6a12159834d048fb89372e58e9729032fe4c214019d6bd6700101dacd"} Nov 03 18:56:36.156822 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:36.156802 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dg7z-trghh" event={"ID":"bcfc2d04-5ae0-4cf1-901b-30b1fac954ee","Type":"ContainerDied","Data":"546a55d253a4c73d266284a4a87caaedeabe0c0f60aab5bcc442678d83ff91f7"} Nov 03 18:56:36.156913 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:36.156832 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="546a55d253a4c73d266284a4a87caaedeabe0c0f60aab5bcc442678d83ff91f7" Nov 03 18:56:36.156913 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:36.156805 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dg7z-trghh" Nov 03 18:56:37.357814 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.357784 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:37.359852 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.359834 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:37.434973 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.434943 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-kube-api-access-kkd7n\" (UniqueName: \"kubernetes.io/projected/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-kube-api-access-kkd7n\") pod \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " Nov 03 18:56:37.435132 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435007 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04889421-f5e1-4a7f-b19d-2da6488c4b8e-kube-api-access-6gmp7\" (UniqueName: \"kubernetes.io/projected/04889421-f5e1-4a7f-b19d-2da6488c4b8e-kube-api-access-6gmp7\") pod \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " Nov 03 18:56:37.435132 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435053 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-openshift-ca\") pod \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " Nov 03 18:56:37.435261 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435186 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-openshift-ca\") pod \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " Nov 03 18:56:37.435261 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435247 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-xks-ca\") pod \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\" (UID: \"3df5c7ca-9b8f-43e8-b283-62238b64cd6b\") " Nov 03 18:56:37.435373 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435306 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-xks-ca\") pod \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\" (UID: \"04889421-f5e1-4a7f-b19d-2da6488c4b8e\") " Nov 03 18:56:37.435754 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435496 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-openshift-ca" pod "3df5c7ca-9b8f-43e8-b283-62238b64cd6b" (UID: "3df5c7ca-9b8f-43e8-b283-62238b64cd6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:37.435754 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435517 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-openshift-ca" pod "04889421-f5e1-4a7f-b19d-2da6488c4b8e" (UID: "04889421-f5e1-4a7f-b19d-2da6488c4b8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:37.435929 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435754 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:37.435929 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.435778 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:37.437458 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.437432 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-xks-ca" pod "04889421-f5e1-4a7f-b19d-2da6488c4b8e" (UID: "04889421-f5e1-4a7f-b19d-2da6488c4b8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:37.437740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.437690 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04889421-f5e1-4a7f-b19d-2da6488c4b8e-kube-api-access-6gmp7" pod "04889421-f5e1-4a7f-b19d-2da6488c4b8e" (UID: "04889421-f5e1-4a7f-b19d-2da6488c4b8e"). InnerVolumeSpecName "kube-api-access-6gmp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:37.437851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.437744 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-kube-api-access-kkd7n" pod "3df5c7ca-9b8f-43e8-b283-62238b64cd6b" (UID: "3df5c7ca-9b8f-43e8-b283-62238b64cd6b"). InnerVolumeSpecName "kube-api-access-kkd7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:37.438019 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.437994 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-xks-ca" pod "3df5c7ca-9b8f-43e8-b283-62238b64cd6b" (UID: "3df5c7ca-9b8f-43e8-b283-62238b64cd6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:37.536381 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.536342 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gmp7\" (UniqueName: \"kubernetes.io/projected/04889421-f5e1-4a7f-b19d-2da6488c4b8e-kube-api-access-6gmp7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:37.536381 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.536374 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:37.536381 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.536384 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04889421-f5e1-4a7f-b19d-2da6488c4b8e-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:37.536613 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:37.536394 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkd7n\" (UniqueName: \"kubernetes.io/projected/3df5c7ca-9b8f-43e8-b283-62238b64cd6b-kube-api-access-kkd7n\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:38.172631 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:38.172603 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" Nov 03 18:56:38.172828 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:38.172636 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk" event={"ID":"04889421-f5e1-4a7f-b19d-2da6488c4b8e","Type":"ContainerDied","Data":"d1b8f5b14a673b0a6fb1c6633a764078073b91e7b630e51264b1f16e8f69f83f"} Nov 03 18:56:38.172828 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:38.172685 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b8f5b14a673b0a6fb1c6633a764078073b91e7b630e51264b1f16e8f69f83f" Nov 03 18:56:38.174239 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:38.174210 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" Nov 03 18:56:38.174348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:38.174257 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v" event={"ID":"3df5c7ca-9b8f-43e8-b283-62238b64cd6b","Type":"ContainerDied","Data":"236319717c291b96015e17a890031af49a5964ec522e6946fa4ebc6f58af7a9a"} Nov 03 18:56:38.174348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:38.174305 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="236319717c291b96015e17a890031af49a5964ec522e6946fa4ebc6f58af7a9a" Nov 03 18:56:39.077498 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.077466 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t"] Nov 03 18:56:39.078042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078025 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3df5c7ca-9b8f-43e8-b283-62238b64cd6b" containerName="revoke-keys" Nov 03 18:56:39.078042 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078042 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df5c7ca-9b8f-43e8-b283-62238b64cd6b" containerName="revoke-keys" Nov 03 18:56:39.078137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078057 4948 cpu_manager.go:414] "containerMap: removing container" podUID="04889421-f5e1-4a7f-b19d-2da6488c4b8e" containerName="revoke-keys" Nov 03 18:56:39.078137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078063 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04889421-f5e1-4a7f-b19d-2da6488c4b8e" containerName="revoke-keys" Nov 03 18:56:39.078137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078084 4948 cpu_manager.go:414] "containerMap: removing container" podUID="bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" containerName="insights-gathering" Nov 03 18:56:39.078137 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078090 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" containerName="insights-gathering" Nov 03 18:56:39.078248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078165 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df5c7ca-9b8f-43e8-b283-62238b64cd6b" containerName="revoke-keys" Nov 03 18:56:39.078248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078175 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="04889421-f5e1-4a7f-b19d-2da6488c4b8e" containerName="revoke-keys" Nov 03 18:56:39.078248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.078184 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfc2d04-5ae0-4cf1-901b-30b1fac954ee" containerName="insights-gathering" Nov 03 18:56:39.081960 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.081938 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.136485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.136453 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t"] Nov 03 18:56:39.152931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.152899 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.153096 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.152980 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x5fk\" (UniqueName: \"kubernetes.io/projected/83ce1ba0-6c27-41a9-899c-737a2083f7bc-kube-api-access-4x5fk\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.153188 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.153156 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.181846 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.181782 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:56:39.182104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.181821 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:56:39.182579 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.182556 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:56:39.186313 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.186293 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:56:39.254084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.254055 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.254248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.254110 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.254248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.254169 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x5fk\" (UniqueName: \"kubernetes.io/projected/83ce1ba0-6c27-41a9-899c-737a2083f7bc-kube-api-access-4x5fk\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.254767 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.254735 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.256674 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.256652 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.319114 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.319079 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x5fk\" (UniqueName: \"kubernetes.io/projected/83ce1ba0-6c27-41a9-899c-737a2083f7bc-kube-api-access-4x5fk\") pod \"maas-api-revoke-sub-e5a4a8f2-fp52t\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.398016 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.397929 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:39.704467 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.704433 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6"] Nov 03 18:56:39.715203 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.715173 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.762855 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.762812 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6"] Nov 03 18:56:39.825085 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.819782 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t"] Nov 03 18:56:39.860071 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.860045 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.860193 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.860077 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfm2p\" (UniqueName: \"kubernetes.io/projected/332895aa-8af7-48b5-a350-aa18b4b300b3-kube-api-access-nfm2p\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.860193 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.860118 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.961560 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.961478 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.961560 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.961529 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfm2p\" (UniqueName: \"kubernetes.io/projected/332895aa-8af7-48b5-a350-aa18b4b300b3-kube-api-access-nfm2p\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.961821 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.961595 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.962162 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.962136 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.963901 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.963878 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:39.983939 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.983916 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m"] Nov 03 18:56:39.989339 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:39.989322 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.023568 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.023538 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfm2p\" (UniqueName: \"kubernetes.io/projected/332895aa-8af7-48b5-a350-aa18b4b300b3-kube-api-access-nfm2p\") pod \"maas-api-revoke-sub-7f5a369b-kpff6\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:40.030349 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.030324 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:40.043141 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.043073 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m"] Nov 03 18:56:40.063138 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.063106 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9t97\" (UniqueName: \"kubernetes.io/projected/44411451-2cd4-4e63-ac95-dc9249b1d2d9-kube-api-access-x9t97\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.063362 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.063342 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.063457 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.063430 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.164650 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.164597 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.165120 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.164731 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.165120 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.164773 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9t97\" (UniqueName: \"kubernetes.io/projected/44411451-2cd4-4e63-ac95-dc9249b1d2d9-kube-api-access-x9t97\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.165427 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.165402 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.167312 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.167282 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.193000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.192972 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" event={"ID":"83ce1ba0-6c27-41a9-899c-737a2083f7bc","Type":"ContainerStarted","Data":"c25c2aeacf79db494e9bebf811ad00c025344598a2e513f75ca65d55f221341d"} Nov 03 18:56:40.227015 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.226953 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9t97\" (UniqueName: \"kubernetes.io/projected/44411451-2cd4-4e63-ac95-dc9249b1d2d9-kube-api-access-x9t97\") pod \"maas-api-revoke-sub-4038e619-2bn9m\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.303839 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.303803 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:40.432828 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.432787 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6"] Nov 03 18:56:40.690068 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:40.690043 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m"] Nov 03 18:56:40.753320 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:40.753246 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44411451_2cd4_4e63_ac95_dc9249b1d2d9.slice/crio-8d4008c446aac4072075cf302f0a7374add453ad7ed2d854737bdd242b0af0d0 WatchSource:0}: Error finding container 8d4008c446aac4072075cf302f0a7374add453ad7ed2d854737bdd242b0af0d0: Status 404 returned error can't find the container with id 8d4008c446aac4072075cf302f0a7374add453ad7ed2d854737bdd242b0af0d0 Nov 03 18:56:41.202102 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.202066 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" event={"ID":"44411451-2cd4-4e63-ac95-dc9249b1d2d9","Type":"ContainerStarted","Data":"42e43a7d9bd2ffd05781496576ae3e99580bf3e54e14d540ec2d0aabfd59fb2c"} Nov 03 18:56:41.202607 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.202137 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" event={"ID":"44411451-2cd4-4e63-ac95-dc9249b1d2d9","Type":"ContainerStarted","Data":"8d4008c446aac4072075cf302f0a7374add453ad7ed2d854737bdd242b0af0d0"} Nov 03 18:56:41.203974 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.203943 4948 generic.go:356] "Generic (PLEG): container finished" podID="83ce1ba0-6c27-41a9-899c-737a2083f7bc" containerID="499077075a67c1111eeaa627b3260358bcb591f8e2b15750c03984461bead9b3" exitCode=0 Nov 03 18:56:41.204093 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.204020 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" event={"ID":"83ce1ba0-6c27-41a9-899c-737a2083f7bc","Type":"ContainerDied","Data":"499077075a67c1111eeaa627b3260358bcb591f8e2b15750c03984461bead9b3"} Nov 03 18:56:41.205521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.205492 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" event={"ID":"332895aa-8af7-48b5-a350-aa18b4b300b3","Type":"ContainerStarted","Data":"cd67cdab18f10050f03451e8401fb2fb949915120e2cfadf0d11baac3fae69ba"} Nov 03 18:56:41.205633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.205531 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" event={"ID":"332895aa-8af7-48b5-a350-aa18b4b300b3","Type":"ContainerStarted","Data":"1a4c3df03f8643a65670de69732ccc3af99cc7bf8ac5a455a084a3e8f87cdf6d"} Nov 03 18:56:41.322931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.322875 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" podStartSLOduration=2.3228560529999998 podStartE2EDuration="2.322856053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:41.319741776 +0000 UTC m=+1130.802737690" watchObservedRunningTime="2026-11-03 18:56:41.322856053 +0000 UTC m=+1130.805851969" Nov 03 18:56:41.556995 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:41.556898 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" podStartSLOduration=2.556882829 podStartE2EDuration="2.556882829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:41.555824416 +0000 UTC m=+1131.038820336" watchObservedRunningTime="2026-11-03 18:56:41.556882829 +0000 UTC m=+1131.039878742" Nov 03 18:56:42.214585 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.214546 4948 generic.go:356] "Generic (PLEG): container finished" podID="332895aa-8af7-48b5-a350-aa18b4b300b3" containerID="cd67cdab18f10050f03451e8401fb2fb949915120e2cfadf0d11baac3fae69ba" exitCode=0 Nov 03 18:56:42.215031 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.214634 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" event={"ID":"332895aa-8af7-48b5-a350-aa18b4b300b3","Type":"ContainerDied","Data":"cd67cdab18f10050f03451e8401fb2fb949915120e2cfadf0d11baac3fae69ba"} Nov 03 18:56:42.216363 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.216328 4948 generic.go:356] "Generic (PLEG): container finished" podID="44411451-2cd4-4e63-ac95-dc9249b1d2d9" containerID="42e43a7d9bd2ffd05781496576ae3e99580bf3e54e14d540ec2d0aabfd59fb2c" exitCode=0 Nov 03 18:56:42.216490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.216383 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" event={"ID":"44411451-2cd4-4e63-ac95-dc9249b1d2d9","Type":"ContainerDied","Data":"42e43a7d9bd2ffd05781496576ae3e99580bf3e54e14d540ec2d0aabfd59fb2c"} Nov 03 18:56:42.382130 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.382103 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:42.495238 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.495149 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-openshift-ca\") pod \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " Nov 03 18:56:42.495387 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.495267 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ce1ba0-6c27-41a9-899c-737a2083f7bc-kube-api-access-4x5fk\" (UniqueName: \"kubernetes.io/projected/83ce1ba0-6c27-41a9-899c-737a2083f7bc-kube-api-access-4x5fk\") pod \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " Nov 03 18:56:42.495387 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.495306 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-xks-ca\") pod \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\" (UID: \"83ce1ba0-6c27-41a9-899c-737a2083f7bc\") " Nov 03 18:56:42.495533 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.495510 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-openshift-ca" pod "83ce1ba0-6c27-41a9-899c-737a2083f7bc" (UID: "83ce1ba0-6c27-41a9-899c-737a2083f7bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:42.495687 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.495663 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:42.497303 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.497275 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-xks-ca" pod "83ce1ba0-6c27-41a9-899c-737a2083f7bc" (UID: "83ce1ba0-6c27-41a9-899c-737a2083f7bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:42.497410 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.497371 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ce1ba0-6c27-41a9-899c-737a2083f7bc-kube-api-access-4x5fk" pod "83ce1ba0-6c27-41a9-899c-737a2083f7bc" (UID: "83ce1ba0-6c27-41a9-899c-737a2083f7bc"). InnerVolumeSpecName "kube-api-access-4x5fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:42.597101 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.597071 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x5fk\" (UniqueName: \"kubernetes.io/projected/83ce1ba0-6c27-41a9-899c-737a2083f7bc-kube-api-access-4x5fk\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:42.597101 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.597099 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ce1ba0-6c27-41a9-899c-737a2083f7bc-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:42.815629 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.815586 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk"] Nov 03 18:56:42.816246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.816226 4948 cpu_manager.go:414] "containerMap: removing container" podUID="83ce1ba0-6c27-41a9-899c-737a2083f7bc" containerName="revoke-keys" Nov 03 18:56:42.816359 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.816248 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ce1ba0-6c27-41a9-899c-737a2083f7bc" containerName="revoke-keys" Nov 03 18:56:42.816464 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.816451 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ce1ba0-6c27-41a9-899c-737a2083f7bc" containerName="revoke-keys" Nov 03 18:56:42.827185 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.827153 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:42.874909 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.874880 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk"] Nov 03 18:56:42.901116 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.901080 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:42.901304 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.901171 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:42.901441 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:42.901415 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7sr\" (UniqueName: \"kubernetes.io/projected/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-kube-api-access-nw7sr\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.002994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.002963 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7sr\" (UniqueName: \"kubernetes.io/projected/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-kube-api-access-nw7sr\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.003190 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.003012 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.003260 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.003197 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.003770 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.003746 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.005452 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.005428 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.066246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.066175 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7sr\" (UniqueName: \"kubernetes.io/projected/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-kube-api-access-nw7sr\") pod \"maas-api-revoke-sub-7a2ef9ce-ftcfk\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.140316 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.140285 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:43.225897 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.225869 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" Nov 03 18:56:43.226316 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.225870 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t" event={"ID":"83ce1ba0-6c27-41a9-899c-737a2083f7bc","Type":"ContainerDied","Data":"c25c2aeacf79db494e9bebf811ad00c025344598a2e513f75ca65d55f221341d"} Nov 03 18:56:43.226316 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.226022 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c25c2aeacf79db494e9bebf811ad00c025344598a2e513f75ca65d55f221341d" Nov 03 18:56:43.398558 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.398526 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:43.436204 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.436178 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:43.512346 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.512306 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332895aa-8af7-48b5-a350-aa18b4b300b3-kube-api-access-nfm2p\" (UniqueName: \"kubernetes.io/projected/332895aa-8af7-48b5-a350-aa18b4b300b3-kube-api-access-nfm2p\") pod \"332895aa-8af7-48b5-a350-aa18b4b300b3\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " Nov 03 18:56:43.512523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.512402 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-xks-ca\") pod \"332895aa-8af7-48b5-a350-aa18b4b300b3\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " Nov 03 18:56:43.512523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.512442 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-xks-ca\") pod \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " Nov 03 18:56:43.512523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.512483 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-openshift-ca\") pod \"332895aa-8af7-48b5-a350-aa18b4b300b3\" (UID: \"332895aa-8af7-48b5-a350-aa18b4b300b3\") " Nov 03 18:56:43.512523 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.512515 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-openshift-ca\") pod \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " Nov 03 18:56:43.512766 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.512585 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44411451-2cd4-4e63-ac95-dc9249b1d2d9-kube-api-access-x9t97\" (UniqueName: \"kubernetes.io/projected/44411451-2cd4-4e63-ac95-dc9249b1d2d9-kube-api-access-x9t97\") pod \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\" (UID: \"44411451-2cd4-4e63-ac95-dc9249b1d2d9\") " Nov 03 18:56:43.512942 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.512910 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-openshift-ca" pod "332895aa-8af7-48b5-a350-aa18b4b300b3" (UID: "332895aa-8af7-48b5-a350-aa18b4b300b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:43.513124 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.513100 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-openshift-ca" pod "44411451-2cd4-4e63-ac95-dc9249b1d2d9" (UID: "44411451-2cd4-4e63-ac95-dc9249b1d2d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:43.513229 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.513210 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:43.513294 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.513233 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:43.515078 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.515048 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332895aa-8af7-48b5-a350-aa18b4b300b3-kube-api-access-nfm2p" pod "332895aa-8af7-48b5-a350-aa18b4b300b3" (UID: "332895aa-8af7-48b5-a350-aa18b4b300b3"). InnerVolumeSpecName "kube-api-access-nfm2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:43.515794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.515770 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-xks-ca" pod "44411451-2cd4-4e63-ac95-dc9249b1d2d9" (UID: "44411451-2cd4-4e63-ac95-dc9249b1d2d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:43.519657 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.518377 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-xks-ca" pod "332895aa-8af7-48b5-a350-aa18b4b300b3" (UID: "332895aa-8af7-48b5-a350-aa18b4b300b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:43.519657 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.518543 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44411451-2cd4-4e63-ac95-dc9249b1d2d9-kube-api-access-x9t97" pod "44411451-2cd4-4e63-ac95-dc9249b1d2d9" (UID: "44411451-2cd4-4e63-ac95-dc9249b1d2d9"). InnerVolumeSpecName "kube-api-access-x9t97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:43.521665 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.521636 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk"] Nov 03 18:56:43.521852 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:43.521830 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc27eb40_4ab7_4b1d_8611_9373b4b4410a.slice/crio-5db04d46390359ef28e5325818915c113fda3e0e69e01e299c6779512096281b WatchSource:0}: Error finding container 5db04d46390359ef28e5325818915c113fda3e0e69e01e299c6779512096281b: Status 404 returned error can't find the container with id 5db04d46390359ef28e5325818915c113fda3e0e69e01e299c6779512096281b Nov 03 18:56:43.614678 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.614647 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfm2p\" (UniqueName: \"kubernetes.io/projected/332895aa-8af7-48b5-a350-aa18b4b300b3-kube-api-access-nfm2p\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:43.614678 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.614675 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/332895aa-8af7-48b5-a350-aa18b4b300b3-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:43.614678 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.614686 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44411451-2cd4-4e63-ac95-dc9249b1d2d9-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:43.614945 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:43.614721 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9t97\" (UniqueName: \"kubernetes.io/projected/44411451-2cd4-4e63-ac95-dc9249b1d2d9-kube-api-access-x9t97\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:44.236229 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.236189 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" event={"ID":"44411451-2cd4-4e63-ac95-dc9249b1d2d9","Type":"ContainerDied","Data":"8d4008c446aac4072075cf302f0a7374add453ad7ed2d854737bdd242b0af0d0"} Nov 03 18:56:44.236672 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.236249 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d4008c446aac4072075cf302f0a7374add453ad7ed2d854737bdd242b0af0d0" Nov 03 18:56:44.236672 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.236202 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m" Nov 03 18:56:44.238275 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.238245 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" event={"ID":"fc27eb40-4ab7-4b1d-8611-9373b4b4410a","Type":"ContainerStarted","Data":"dd311bf6521f27fbf8d4191800e44b99a3d7e6f960b2cf81d9d4d42a7f2b7ec3"} Nov 03 18:56:44.238422 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.238297 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" event={"ID":"fc27eb40-4ab7-4b1d-8611-9373b4b4410a","Type":"ContainerStarted","Data":"5db04d46390359ef28e5325818915c113fda3e0e69e01e299c6779512096281b"} Nov 03 18:56:44.239921 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.239900 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" event={"ID":"332895aa-8af7-48b5-a350-aa18b4b300b3","Type":"ContainerDied","Data":"1a4c3df03f8643a65670de69732ccc3af99cc7bf8ac5a455a084a3e8f87cdf6d"} Nov 03 18:56:44.239921 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.239910 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6" Nov 03 18:56:44.240086 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.239937 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4c3df03f8643a65670de69732ccc3af99cc7bf8ac5a455a084a3e8f87cdf6d" Nov 03 18:56:44.367504 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:44.367441 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" podStartSLOduration=2.367422869 podStartE2EDuration="2.367422869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:44.361763332 +0000 UTC m=+1133.844759244" watchObservedRunningTime="2026-11-03 18:56:44.367422869 +0000 UTC m=+1133.850418784" Nov 03 18:56:45.248882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:45.248850 4948 generic.go:356] "Generic (PLEG): container finished" podID="fc27eb40-4ab7-4b1d-8611-9373b4b4410a" containerID="dd311bf6521f27fbf8d4191800e44b99a3d7e6f960b2cf81d9d4d42a7f2b7ec3" exitCode=0 Nov 03 18:56:45.249264 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:45.248927 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" event={"ID":"fc27eb40-4ab7-4b1d-8611-9373b4b4410a","Type":"ContainerDied","Data":"dd311bf6521f27fbf8d4191800e44b99a3d7e6f960b2cf81d9d4d42a7f2b7ec3"} Nov 03 18:56:46.414272 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.414245 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:46.547315 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.547221 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-openshift-ca\") pod \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " Nov 03 18:56:46.547315 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.547283 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-kube-api-access-nw7sr\" (UniqueName: \"kubernetes.io/projected/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-kube-api-access-nw7sr\") pod \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " Nov 03 18:56:46.547541 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.547395 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-xks-ca\") pod \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\" (UID: \"fc27eb40-4ab7-4b1d-8611-9373b4b4410a\") " Nov 03 18:56:46.547709 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.547654 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-openshift-ca" pod "fc27eb40-4ab7-4b1d-8611-9373b4b4410a" (UID: "fc27eb40-4ab7-4b1d-8611-9373b4b4410a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:46.547999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.547974 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:46.549527 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.549492 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-xks-ca" pod "fc27eb40-4ab7-4b1d-8611-9373b4b4410a" (UID: "fc27eb40-4ab7-4b1d-8611-9373b4b4410a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:46.549768 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.549751 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-kube-api-access-nw7sr" pod "fc27eb40-4ab7-4b1d-8611-9373b4b4410a" (UID: "fc27eb40-4ab7-4b1d-8611-9373b4b4410a"). InnerVolumeSpecName "kube-api-access-nw7sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:46.648778 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.648745 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7sr\" (UniqueName: \"kubernetes.io/projected/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-kube-api-access-nw7sr\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:46.648778 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:46.648773 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27eb40-4ab7-4b1d-8611-9373b4b4410a-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:47.303429 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:47.303388 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" event={"ID":"fc27eb40-4ab7-4b1d-8611-9373b4b4410a","Type":"ContainerDied","Data":"5db04d46390359ef28e5325818915c113fda3e0e69e01e299c6779512096281b"} Nov 03 18:56:47.303902 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:47.303733 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk" Nov 03 18:56:47.304085 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:47.303889 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db04d46390359ef28e5325818915c113fda3e0e69e01e299c6779512096281b" Nov 03 18:56:48.462089 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.462049 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg"] Nov 03 18:56:48.463335 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463308 4948 cpu_manager.go:414] "containerMap: removing container" podUID="332895aa-8af7-48b5-a350-aa18b4b300b3" containerName="revoke-keys" Nov 03 18:56:48.463335 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463333 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332895aa-8af7-48b5-a350-aa18b4b300b3" containerName="revoke-keys" Nov 03 18:56:48.463520 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463347 4948 cpu_manager.go:414] "containerMap: removing container" podUID="fc27eb40-4ab7-4b1d-8611-9373b4b4410a" containerName="revoke-keys" Nov 03 18:56:48.463520 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463355 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc27eb40-4ab7-4b1d-8611-9373b4b4410a" containerName="revoke-keys" Nov 03 18:56:48.463520 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463390 4948 cpu_manager.go:414] "containerMap: removing container" podUID="44411451-2cd4-4e63-ac95-dc9249b1d2d9" containerName="revoke-keys" Nov 03 18:56:48.463520 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463397 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44411451-2cd4-4e63-ac95-dc9249b1d2d9" containerName="revoke-keys" Nov 03 18:56:48.463740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463605 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc27eb40-4ab7-4b1d-8611-9373b4b4410a" containerName="revoke-keys" Nov 03 18:56:48.463740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463626 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="332895aa-8af7-48b5-a350-aa18b4b300b3" containerName="revoke-keys" Nov 03 18:56:48.463740 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.463640 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="44411451-2cd4-4e63-ac95-dc9249b1d2d9" containerName="revoke-keys" Nov 03 18:56:48.467645 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.467614 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.518191 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.518159 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg"] Nov 03 18:56:48.531653 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.531622 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"openshift-service-ca.crt\"" Nov 03 18:56:48.565072 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.565049 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"kube-root-ca.crt\"" Nov 03 18:56:48.565229 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.565139 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"distinct-w3-a4cacd-kserve-self-signed-certs\"" Nov 03 18:56:48.574487 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.574467 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"default-dockercfg-vzdn4\"" Nov 03 18:56:48.609263 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.609230 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-dshm\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.609570 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.609544 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-model-cache\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.609742 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.609722 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tls-certs\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.609920 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.609899 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-home\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.610130 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.610114 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tmp-dir\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.610277 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.610262 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2j24\" (UniqueName: \"kubernetes.io/projected/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-kube-api-access-t2j24\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712008 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.711973 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-dshm\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712237 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712075 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-model-cache\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712237 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712129 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tls-certs\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712237 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712183 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-home\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712419 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712243 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tmp-dir\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712419 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712294 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2j24\" (UniqueName: \"kubernetes.io/projected/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-kube-api-access-t2j24\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712533 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712465 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-model-cache\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712533 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712500 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-home\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.712990 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.712965 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tmp-dir\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.714932 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.714905 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-dshm\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.715358 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.715330 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tls-certs\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.774618 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.774586 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2j24\" (UniqueName: \"kubernetes.io/projected/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-kube-api-access-t2j24\") pod \"distinct-w3-a4cacd-kserve-5b4d546576-nnljg\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:48.786940 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:48.786912 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:49.236221 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:49.236184 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d3f7d8c_ba02_44e3_90fe_97c5c8206bec.slice/crio-5f09b21ad1838476ccde9f939bc6c96ffb9681518df065cd7045f3701b9bcf32 WatchSource:0}: Error finding container 5f09b21ad1838476ccde9f939bc6c96ffb9681518df065cd7045f3701b9bcf32: Status 404 returned error can't find the container with id 5f09b21ad1838476ccde9f939bc6c96ffb9681518df065cd7045f3701b9bcf32 Nov 03 18:56:49.236431 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:49.236411 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg"] Nov 03 18:56:49.324045 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:49.324008 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" event={"ID":"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec","Type":"ContainerStarted","Data":"279ba8cf87e577c77c219a583eb804205828a18b6f5e851176f0a7a24d64d26a"} Nov 03 18:56:49.324045 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:49.324052 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" event={"ID":"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec","Type":"ContainerStarted","Data":"5f09b21ad1838476ccde9f939bc6c96ffb9681518df065cd7045f3701b9bcf32"} Nov 03 18:56:49.462324 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:49.462275 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podStartSLOduration=1.462261144 podStartE2EDuration="1.462261144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:49.458914185 +0000 UTC m=+1138.941910098" watchObservedRunningTime="2026-11-03 18:56:49.462261144 +0000 UTC m=+1138.945257058" Nov 03 18:56:49.787036 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:49.787004 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:50.482566 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.482530 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d"] Nov 03 18:56:50.487623 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.487600 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.539961 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.539934 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d"] Nov 03 18:56:50.585783 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.585749 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:56:50.585965 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.585839 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:56:50.586074 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.586057 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:56:50.592111 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.592087 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:56:50.646381 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.646343 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.646529 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.646398 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4q8\" (UniqueName: \"kubernetes.io/projected/2bef0166-1c95-4f78-88e3-e098e1343d1e-kube-api-access-zm4q8\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.646589 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.646552 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.747254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.747158 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4q8\" (UniqueName: \"kubernetes.io/projected/2bef0166-1c95-4f78-88e3-e098e1343d1e-kube-api-access-zm4q8\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.747422 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.747253 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.747422 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.747374 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.799875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.799846 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:56:50.800038 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.799885 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:56:50.808974 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.808942 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.810723 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.810677 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:50.859807 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.859774 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:56:50.871281 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:50.871251 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4q8\" (UniqueName: \"kubernetes.io/projected/2bef0166-1c95-4f78-88e3-e098e1343d1e-kube-api-access-zm4q8\") pod \"maas-api-revoke-sub-e5de52ce-xfs4d\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:51.156391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.156306 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:56:51.164468 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.164426 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:51.569758 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.555468 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d"] Nov 03 18:56:51.738683 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.738653 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp"] Nov 03 18:56:51.743931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.743912 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.757951 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.757854 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4lnz\" (UniqueName: \"kubernetes.io/projected/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-kube-api-access-r4lnz\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.758639 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.758202 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.758760 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.758741 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.798817 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.798786 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp"] Nov 03 18:56:51.859932 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.859865 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4lnz\" (UniqueName: \"kubernetes.io/projected/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-kube-api-access-r4lnz\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.860084 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.860057 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.860215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.860128 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.860892 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.860868 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.863374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.863350 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:51.921965 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:51.921935 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4lnz\" (UniqueName: \"kubernetes.io/projected/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-kube-api-access-r4lnz\") pod \"maas-api-revoke-sub-b078fad8-689qp\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:52.059568 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:52.059532 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:52.350302 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:52.350270 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" event={"ID":"2bef0166-1c95-4f78-88e3-e098e1343d1e","Type":"ContainerStarted","Data":"1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8"} Nov 03 18:56:52.350448 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:52.350318 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" event={"ID":"2bef0166-1c95-4f78-88e3-e098e1343d1e","Type":"ContainerStarted","Data":"90942f78b52a37acb3c6526c4f45326cfc5d980e65ed607641fa03ea62096b30"} Nov 03 18:56:52.452262 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:52.452229 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd1ba67_b3d2_461a_af8a_d6d030af1a27.slice/crio-8b4200f3f90d6ddd385f124d53fe46f278a615fe92fd4c04b81bd683a42ff1fd WatchSource:0}: Error finding container 8b4200f3f90d6ddd385f124d53fe46f278a615fe92fd4c04b81bd683a42ff1fd: Status 404 returned error can't find the container with id 8b4200f3f90d6ddd385f124d53fe46f278a615fe92fd4c04b81bd683a42ff1fd Nov 03 18:56:52.453507 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:52.453482 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp"] Nov 03 18:56:52.472306 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:52.472248 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" podStartSLOduration=2.472228823 podStartE2EDuration="2.472228823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:52.46766442 +0000 UTC m=+1141.950660335" watchObservedRunningTime="2026-11-03 18:56:52.472228823 +0000 UTC m=+1141.955224738" Nov 03 18:56:53.360309 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:53.360271 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" event={"ID":"2fd1ba67-b3d2-461a-af8a-d6d030af1a27","Type":"ContainerStarted","Data":"61a62d03786067fd672ead3076b95f9a2d974ed1991ed5c82c6d9956dd181ae7"} Nov 03 18:56:53.360866 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:53.360337 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" event={"ID":"2fd1ba67-b3d2-461a-af8a-d6d030af1a27","Type":"ContainerStarted","Data":"8b4200f3f90d6ddd385f124d53fe46f278a615fe92fd4c04b81bd683a42ff1fd"} Nov 03 18:56:53.362264 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:53.362238 4948 generic.go:356] "Generic (PLEG): container finished" podID="2bef0166-1c95-4f78-88e3-e098e1343d1e" containerID="1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8" exitCode=0 Nov 03 18:56:53.362387 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:53.362321 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" event={"ID":"2bef0166-1c95-4f78-88e3-e098e1343d1e","Type":"ContainerDied","Data":"1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8"} Nov 03 18:56:53.485391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:53.485337 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" podStartSLOduration=2.485323014 podStartE2EDuration="2.485323014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:53.481723848 +0000 UTC m=+1142.964719763" watchObservedRunningTime="2026-11-03 18:56:53.485323014 +0000 UTC m=+1142.968318928" Nov 03 18:56:54.371837 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.371802 4948 generic.go:356] "Generic (PLEG): container finished" podID="2fd1ba67-b3d2-461a-af8a-d6d030af1a27" containerID="61a62d03786067fd672ead3076b95f9a2d974ed1991ed5c82c6d9956dd181ae7" exitCode=0 Nov 03 18:56:54.372215 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.371893 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" event={"ID":"2fd1ba67-b3d2-461a-af8a-d6d030af1a27","Type":"ContainerDied","Data":"61a62d03786067fd672ead3076b95f9a2d974ed1991ed5c82c6d9956dd181ae7"} Nov 03 18:56:54.540636 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.540611 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:54.592075 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.592038 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-openshift-ca\") pod \"2bef0166-1c95-4f78-88e3-e098e1343d1e\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " Nov 03 18:56:54.592247 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.592123 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bef0166-1c95-4f78-88e3-e098e1343d1e-kube-api-access-zm4q8\" (UniqueName: \"kubernetes.io/projected/2bef0166-1c95-4f78-88e3-e098e1343d1e-kube-api-access-zm4q8\") pod \"2bef0166-1c95-4f78-88e3-e098e1343d1e\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " Nov 03 18:56:54.592247 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.592186 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-xks-ca\") pod \"2bef0166-1c95-4f78-88e3-e098e1343d1e\" (UID: \"2bef0166-1c95-4f78-88e3-e098e1343d1e\") " Nov 03 18:56:54.592485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.592459 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-openshift-ca" pod "2bef0166-1c95-4f78-88e3-e098e1343d1e" (UID: "2bef0166-1c95-4f78-88e3-e098e1343d1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:54.594317 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.594293 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bef0166-1c95-4f78-88e3-e098e1343d1e-kube-api-access-zm4q8" pod "2bef0166-1c95-4f78-88e3-e098e1343d1e" (UID: "2bef0166-1c95-4f78-88e3-e098e1343d1e"). InnerVolumeSpecName "kube-api-access-zm4q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:54.594429 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.594407 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-xks-ca" pod "2bef0166-1c95-4f78-88e3-e098e1343d1e" (UID: "2bef0166-1c95-4f78-88e3-e098e1343d1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:54.693780 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.693748 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:54.693780 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.693783 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4q8\" (UniqueName: \"kubernetes.io/projected/2bef0166-1c95-4f78-88e3-e098e1343d1e-kube-api-access-zm4q8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:54.694003 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:54.693800 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bef0166-1c95-4f78-88e3-e098e1343d1e-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:55.380887 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.380854 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" Nov 03 18:56:55.381276 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.380895 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d" event={"ID":"2bef0166-1c95-4f78-88e3-e098e1343d1e","Type":"ContainerDied","Data":"90942f78b52a37acb3c6526c4f45326cfc5d980e65ed607641fa03ea62096b30"} Nov 03 18:56:55.381276 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.380935 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90942f78b52a37acb3c6526c4f45326cfc5d980e65ed607641fa03ea62096b30" Nov 03 18:56:55.549159 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.549131 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:55.603858 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.603825 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-xks-ca\") pod \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " Nov 03 18:56:55.603999 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.603878 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-openshift-ca\") pod \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " Nov 03 18:56:55.604037 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.604012 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-kube-api-access-r4lnz\" (UniqueName: \"kubernetes.io/projected/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-kube-api-access-r4lnz\") pod \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\" (UID: \"2fd1ba67-b3d2-461a-af8a-d6d030af1a27\") " Nov 03 18:56:55.604246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.604218 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-openshift-ca" pod "2fd1ba67-b3d2-461a-af8a-d6d030af1a27" (UID: "2fd1ba67-b3d2-461a-af8a-d6d030af1a27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:55.604407 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.604390 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:55.605963 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.605936 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-xks-ca" pod "2fd1ba67-b3d2-461a-af8a-d6d030af1a27" (UID: "2fd1ba67-b3d2-461a-af8a-d6d030af1a27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:55.606098 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.606080 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-kube-api-access-r4lnz" pod "2fd1ba67-b3d2-461a-af8a-d6d030af1a27" (UID: "2fd1ba67-b3d2-461a-af8a-d6d030af1a27"). InnerVolumeSpecName "kube-api-access-r4lnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:55.705403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.705372 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4lnz\" (UniqueName: \"kubernetes.io/projected/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-kube-api-access-r4lnz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:55.705403 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:55.705401 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd1ba67-b3d2-461a-af8a-d6d030af1a27-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:56.149890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.149798 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq"] Nov 03 18:56:56.150714 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.150682 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2fd1ba67-b3d2-461a-af8a-d6d030af1a27" containerName="revoke-keys" Nov 03 18:56:56.150835 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.150718 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd1ba67-b3d2-461a-af8a-d6d030af1a27" containerName="revoke-keys" Nov 03 18:56:56.150835 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.150732 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2bef0166-1c95-4f78-88e3-e098e1343d1e" containerName="revoke-keys" Nov 03 18:56:56.150835 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.150741 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bef0166-1c95-4f78-88e3-e098e1343d1e" containerName="revoke-keys" Nov 03 18:56:56.150989 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.150904 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd1ba67-b3d2-461a-af8a-d6d030af1a27" containerName="revoke-keys" Nov 03 18:56:56.150989 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.150918 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bef0166-1c95-4f78-88e3-e098e1343d1e" containerName="revoke-keys" Nov 03 18:56:56.162466 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.162421 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.205398 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.205367 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq"] Nov 03 18:56:56.210882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.210851 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ththw\" (UniqueName: \"kubernetes.io/projected/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-kube-api-access-ththw\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.211049 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.211016 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.211149 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.211131 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.311950 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.311914 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ththw\" (UniqueName: \"kubernetes.io/projected/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-kube-api-access-ththw\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.312110 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.311997 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.312110 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.312045 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.312690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.312668 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.314532 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.314502 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.374062 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.374030 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ththw\" (UniqueName: \"kubernetes.io/projected/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-kube-api-access-ththw\") pod \"maas-api-revoke-sub-80ae7e51-gmhwq\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.392025 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.391999 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" Nov 03 18:56:56.392421 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.392039 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp" event={"ID":"2fd1ba67-b3d2-461a-af8a-d6d030af1a27","Type":"ContainerDied","Data":"8b4200f3f90d6ddd385f124d53fe46f278a615fe92fd4c04b81bd683a42ff1fd"} Nov 03 18:56:56.392421 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.392084 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b4200f3f90d6ddd385f124d53fe46f278a615fe92fd4c04b81bd683a42ff1fd" Nov 03 18:56:56.477832 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.477801 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:56.607391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.607349 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq"] Nov 03 18:56:56.612947 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.612924 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.665067 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.665040 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq"] Nov 03 18:56:56.716643 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.716611 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.716816 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.716786 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.716816 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.716811 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cz5t\" (UniqueName: \"kubernetes.io/projected/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-kube-api-access-6cz5t\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.817759 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.817660 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.817906 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.817812 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.817906 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.817847 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cz5t\" (UniqueName: \"kubernetes.io/projected/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-kube-api-access-6cz5t\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.818440 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.818418 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.820094 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.820068 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.868108 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:56.868080 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9c5ad5_8c1e_4962_96c7_15f77188bd48.slice/crio-219d776cfeeb59e303fb677953f94f8742ccc28f6e1b72b590a16d3fef1e0cda WatchSource:0}: Error finding container 219d776cfeeb59e303fb677953f94f8742ccc28f6e1b72b590a16d3fef1e0cda: Status 404 returned error can't find the container with id 219d776cfeeb59e303fb677953f94f8742ccc28f6e1b72b590a16d3fef1e0cda Nov 03 18:56:56.869122 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.869084 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq"] Nov 03 18:56:56.879960 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.879937 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cz5t\" (UniqueName: \"kubernetes.io/projected/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-kube-api-access-6cz5t\") pod \"maas-api-revoke-sub-65e61e90-vsgmq\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:56.928496 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:56.928473 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:57.305746 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:57.305688 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d4eb00_ff37_4ca4_b3ae_3bb445c9cf63.slice/crio-874412e29695092dd28d8cae0ea4ec105276b72b43288a5955d6728f739d2682 WatchSource:0}: Error finding container 874412e29695092dd28d8cae0ea4ec105276b72b43288a5955d6728f739d2682: Status 404 returned error can't find the container with id 874412e29695092dd28d8cae0ea4ec105276b72b43288a5955d6728f739d2682 Nov 03 18:56:57.305965 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:57.305941 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq"] Nov 03 18:56:57.403657 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:57.403622 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" event={"ID":"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63","Type":"ContainerStarted","Data":"874412e29695092dd28d8cae0ea4ec105276b72b43288a5955d6728f739d2682"} Nov 03 18:56:57.405465 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:57.405435 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" event={"ID":"8d9c5ad5-8c1e-4962-96c7-15f77188bd48","Type":"ContainerStarted","Data":"a478a99c0c3faf7871520a7712c764023551699d98572d9cdd60a2ef86113183"} Nov 03 18:56:57.405591 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:57.405482 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" event={"ID":"8d9c5ad5-8c1e-4962-96c7-15f77188bd48","Type":"ContainerStarted","Data":"219d776cfeeb59e303fb677953f94f8742ccc28f6e1b72b590a16d3fef1e0cda"} Nov 03 18:56:57.526139 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:57.526091 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" podStartSLOduration=1.526078618 podStartE2EDuration="1.526078618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:57.522536577 +0000 UTC m=+1147.005532472" watchObservedRunningTime="2026-11-03 18:56:57.526078618 +0000 UTC m=+1147.009074541" Nov 03 18:56:58.418239 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.418203 4948 generic.go:356] "Generic (PLEG): container finished" podID="49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" containerID="6e64c05e3cbc2be8b480ca6ef888e0b71d75a94666b0f8962af14fc93410e62e" exitCode=0 Nov 03 18:56:58.418842 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.418294 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" event={"ID":"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63","Type":"ContainerDied","Data":"6e64c05e3cbc2be8b480ca6ef888e0b71d75a94666b0f8962af14fc93410e62e"} Nov 03 18:56:58.420254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.420229 4948 generic.go:356] "Generic (PLEG): container finished" podID="8d9c5ad5-8c1e-4962-96c7-15f77188bd48" containerID="a478a99c0c3faf7871520a7712c764023551699d98572d9cdd60a2ef86113183" exitCode=0 Nov 03 18:56:58.420377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.420313 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" event={"ID":"8d9c5ad5-8c1e-4962-96c7-15f77188bd48","Type":"ContainerDied","Data":"a478a99c0c3faf7871520a7712c764023551699d98572d9cdd60a2ef86113183"} Nov 03 18:56:58.618790 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.618690 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4"] Nov 03 18:56:58.623192 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.623169 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.678248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.678213 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4"] Nov 03 18:56:58.737174 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.737140 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.737347 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.737301 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc22t\" (UniqueName: \"kubernetes.io/projected/0d07c08e-d01a-45b1-83b9-4e6032930224-kube-api-access-qc22t\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.737415 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.737368 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.787338 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.787306 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:58.800587 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.800567 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:58.838914 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.838884 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.839193 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.839165 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.839355 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.839335 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc22t\" (UniqueName: \"kubernetes.io/projected/0d07c08e-d01a-45b1-83b9-4e6032930224-kube-api-access-qc22t\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.839780 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.839758 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.841526 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.841508 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.901846 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.901773 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc22t\" (UniqueName: \"kubernetes.io/projected/0d07c08e-d01a-45b1-83b9-4e6032930224-kube-api-access-qc22t\") pod \"maas-api-revoke-sub-17e56449-lthf4\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:58.937842 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:58.937815 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:56:59.319539 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:56:59.319512 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d07c08e_d01a_45b1_83b9_4e6032930224.slice/crio-5341ca24303fecceb6c7010bacba4b1d9c098e51f4ef2d3ca73722018a3e077b WatchSource:0}: Error finding container 5341ca24303fecceb6c7010bacba4b1d9c098e51f4ef2d3ca73722018a3e077b: Status 404 returned error can't find the container with id 5341ca24303fecceb6c7010bacba4b1d9c098e51f4ef2d3ca73722018a3e077b Nov 03 18:56:59.319686 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.319663 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4"] Nov 03 18:56:59.429490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.429454 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" event={"ID":"0d07c08e-d01a-45b1-83b9-4e6032930224","Type":"ContainerStarted","Data":"5341ca24303fecceb6c7010bacba4b1d9c098e51f4ef2d3ca73722018a3e077b"} Nov 03 18:56:59.442410 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.442386 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 18:56:59.597685 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.597660 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:56:59.641711 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.641684 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:56:59.670639 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.670609 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-openshift-ca\") pod \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " Nov 03 18:56:59.670797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.670673 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-kube-api-access-6cz5t\" (UniqueName: \"kubernetes.io/projected/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-kube-api-access-6cz5t\") pod \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " Nov 03 18:56:59.670797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.670765 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-xks-ca\") pod \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\" (UID: \"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63\") " Nov 03 18:56:59.671044 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.671017 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-openshift-ca" pod "49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" (UID: "49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:59.671246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.671227 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:59.673311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.672926 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-kube-api-access-6cz5t" pod "49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" (UID: "49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63"). InnerVolumeSpecName "kube-api-access-6cz5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:59.673311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.673025 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-xks-ca" pod "49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" (UID: "49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:59.771987 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.771963 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-xks-ca\") pod \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " Nov 03 18:56:59.772183 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.772168 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-openshift-ca\") pod \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " Nov 03 18:56:59.772246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.772210 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-kube-api-access-ththw\" (UniqueName: \"kubernetes.io/projected/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-kube-api-access-ththw\") pod \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\" (UID: \"8d9c5ad5-8c1e-4962-96c7-15f77188bd48\") " Nov 03 18:56:59.772536 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.772510 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-openshift-ca" pod "8d9c5ad5-8c1e-4962-96c7-15f77188bd48" (UID: "8d9c5ad5-8c1e-4962-96c7-15f77188bd48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:56:59.772623 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.772588 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cz5t\" (UniqueName: \"kubernetes.io/projected/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-kube-api-access-6cz5t\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:59.772623 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.772610 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:59.773908 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.773886 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-xks-ca" pod "8d9c5ad5-8c1e-4962-96c7-15f77188bd48" (UID: "8d9c5ad5-8c1e-4962-96c7-15f77188bd48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:56:59.774236 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.774213 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-kube-api-access-ththw" pod "8d9c5ad5-8c1e-4962-96c7-15f77188bd48" (UID: "8d9c5ad5-8c1e-4962-96c7-15f77188bd48"). InnerVolumeSpecName "kube-api-access-ththw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:56:59.873679 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.873645 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:59.873854 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.873682 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ththw\" (UniqueName: \"kubernetes.io/projected/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-kube-api-access-ththw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:56:59.873854 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:56:59.873720 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9c5ad5-8c1e-4962-96c7-15f77188bd48-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:00.440546 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.440508 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" event={"ID":"0d07c08e-d01a-45b1-83b9-4e6032930224","Type":"ContainerStarted","Data":"261aa9ab6ed72ece0cd33eb2507b19688bcc5fd67d5b93eaac06d9eb661eaf74"} Nov 03 18:57:00.442294 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.442273 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" Nov 03 18:57:00.442294 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.442289 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq" event={"ID":"49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63","Type":"ContainerDied","Data":"874412e29695092dd28d8cae0ea4ec105276b72b43288a5955d6728f739d2682"} Nov 03 18:57:00.442476 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.442324 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="874412e29695092dd28d8cae0ea4ec105276b72b43288a5955d6728f739d2682" Nov 03 18:57:00.443968 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.443946 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" Nov 03 18:57:00.444076 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.443978 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq" event={"ID":"8d9c5ad5-8c1e-4962-96c7-15f77188bd48","Type":"ContainerDied","Data":"219d776cfeeb59e303fb677953f94f8742ccc28f6e1b72b590a16d3fef1e0cda"} Nov 03 18:57:00.444076 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.444032 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="219d776cfeeb59e303fb677953f94f8742ccc28f6e1b72b590a16d3fef1e0cda" Nov 03 18:57:00.561379 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:00.561330 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" podStartSLOduration=2.5613166290000002 podStartE2EDuration="2.561316629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:00.558231462 +0000 UTC m=+1150.041227377" watchObservedRunningTime="2026-11-03 18:57:00.561316629 +0000 UTC m=+1150.044312542" Nov 03 18:57:01.453994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:01.453961 4948 generic.go:356] "Generic (PLEG): container finished" podID="0d07c08e-d01a-45b1-83b9-4e6032930224" containerID="261aa9ab6ed72ece0cd33eb2507b19688bcc5fd67d5b93eaac06d9eb661eaf74" exitCode=0 Nov 03 18:57:01.454495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:01.454041 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" event={"ID":"0d07c08e-d01a-45b1-83b9-4e6032930224","Type":"ContainerDied","Data":"261aa9ab6ed72ece0cd33eb2507b19688bcc5fd67d5b93eaac06d9eb661eaf74"} Nov 03 18:57:02.612099 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.612075 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:57:02.705648 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.705616 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d07c08e-d01a-45b1-83b9-4e6032930224-kube-api-access-qc22t\" (UniqueName: \"kubernetes.io/projected/0d07c08e-d01a-45b1-83b9-4e6032930224-kube-api-access-qc22t\") pod \"0d07c08e-d01a-45b1-83b9-4e6032930224\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " Nov 03 18:57:02.705933 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.705674 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-xks-ca\") pod \"0d07c08e-d01a-45b1-83b9-4e6032930224\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " Nov 03 18:57:02.705933 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.705735 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-openshift-ca\") pod \"0d07c08e-d01a-45b1-83b9-4e6032930224\" (UID: \"0d07c08e-d01a-45b1-83b9-4e6032930224\") " Nov 03 18:57:02.706334 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.706304 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-openshift-ca" pod "0d07c08e-d01a-45b1-83b9-4e6032930224" (UID: "0d07c08e-d01a-45b1-83b9-4e6032930224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:57:02.707912 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.707879 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d07c08e-d01a-45b1-83b9-4e6032930224-kube-api-access-qc22t" pod "0d07c08e-d01a-45b1-83b9-4e6032930224" (UID: "0d07c08e-d01a-45b1-83b9-4e6032930224"). InnerVolumeSpecName "kube-api-access-qc22t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:02.708001 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.707884 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-xks-ca" pod "0d07c08e-d01a-45b1-83b9-4e6032930224" (UID: "0d07c08e-d01a-45b1-83b9-4e6032930224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:57:02.807840 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.807421 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc22t\" (UniqueName: \"kubernetes.io/projected/0d07c08e-d01a-45b1-83b9-4e6032930224-kube-api-access-qc22t\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:02.807840 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.807455 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:02.807840 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:02.807469 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d07c08e-d01a-45b1-83b9-4e6032930224-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:02.824678 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:57:02.824643 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:57:03.471915 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:03.471805 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" Nov 03 18:57:03.471915 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:03.471841 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4" event={"ID":"0d07c08e-d01a-45b1-83b9-4e6032930224","Type":"ContainerDied","Data":"5341ca24303fecceb6c7010bacba4b1d9c098e51f4ef2d3ca73722018a3e077b"} Nov 03 18:57:03.471915 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:03.471907 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5341ca24303fecceb6c7010bacba4b1d9c098e51f4ef2d3ca73722018a3e077b" Nov 03 18:57:06.596942 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.596902 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9"] Nov 03 18:57:06.597802 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.597780 4948 cpu_manager.go:414] "containerMap: removing container" podUID="0d07c08e-d01a-45b1-83b9-4e6032930224" containerName="revoke-keys" Nov 03 18:57:06.597802 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.597802 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d07c08e-d01a-45b1-83b9-4e6032930224" containerName="revoke-keys" Nov 03 18:57:06.598004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.597817 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8d9c5ad5-8c1e-4962-96c7-15f77188bd48" containerName="revoke-keys" Nov 03 18:57:06.598004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.597825 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9c5ad5-8c1e-4962-96c7-15f77188bd48" containerName="revoke-keys" Nov 03 18:57:06.598004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.597836 4948 cpu_manager.go:414] "containerMap: removing container" podUID="49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" containerName="revoke-keys" Nov 03 18:57:06.598004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.597844 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" containerName="revoke-keys" Nov 03 18:57:06.598004 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.598000 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d07c08e-d01a-45b1-83b9-4e6032930224" containerName="revoke-keys" Nov 03 18:57:06.598254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.598011 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9c5ad5-8c1e-4962-96c7-15f77188bd48" containerName="revoke-keys" Nov 03 18:57:06.598254 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.598021 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" containerName="revoke-keys" Nov 03 18:57:06.602037 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.602018 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.676752 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.653921 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9"] Nov 03 18:57:06.700741 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.698918 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:57:06.700741 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.698927 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:57:06.700741 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.699027 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:57:06.700741 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.699135 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:57:06.747342 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.747313 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfhv\" (UniqueName: \"kubernetes.io/projected/4ada7914-3437-4c4e-b39f-04610941e91f-kube-api-access-4wfhv\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.747497 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.747451 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.747539 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.747495 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.849091 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.849005 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.849091 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.849047 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.849317 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.849135 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfhv\" (UniqueName: \"kubernetes.io/projected/4ada7914-3437-4c4e-b39f-04610941e91f-kube-api-access-4wfhv\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.858259 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.850192 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.858259 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.857824 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.912483 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.912452 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfhv\" (UniqueName: \"kubernetes.io/projected/4ada7914-3437-4c4e-b39f-04610941e91f-kube-api-access-4wfhv\") pod \"maas-api-revoke-sub-722beb94-t62s9\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.915169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.915148 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:06.998753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:06.996141 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87"] Nov 03 18:57:07.002872 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.002533 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.046524 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.046488 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87"] Nov 03 18:57:07.154191 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.154087 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.154191 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.154133 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswj8\" (UniqueName: \"kubernetes.io/projected/641376a9-6fb0-480e-bee1-f48c41b4ac89-kube-api-access-sswj8\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.154425 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.154318 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.255376 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.255339 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.255553 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.255452 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.255553 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.255479 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswj8\" (UniqueName: \"kubernetes.io/projected/641376a9-6fb0-480e-bee1-f48c41b4ac89-kube-api-access-sswj8\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.256246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.256216 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.257914 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.257884 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.299351 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:57:07.299323 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ada7914_3437_4c4e_b39f_04610941e91f.slice/crio-9987f1ce7341056ceb9f5fe48f8871bd6f3ae4056a52d94835635d437020c269 WatchSource:0}: Error finding container 9987f1ce7341056ceb9f5fe48f8871bd6f3ae4056a52d94835635d437020c269: Status 404 returned error can't find the container with id 9987f1ce7341056ceb9f5fe48f8871bd6f3ae4056a52d94835635d437020c269 Nov 03 18:57:07.299976 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.299950 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9"] Nov 03 18:57:07.318000 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.317978 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswj8\" (UniqueName: \"kubernetes.io/projected/641376a9-6fb0-480e-bee1-f48c41b4ac89-kube-api-access-sswj8\") pod \"maas-api-revoke-sub-5161eb69-fnv87\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.321942 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.321920 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:07.510531 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.509935 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" event={"ID":"4ada7914-3437-4c4e-b39f-04610941e91f","Type":"ContainerStarted","Data":"9987f1ce7341056ceb9f5fe48f8871bd6f3ae4056a52d94835635d437020c269"} Nov 03 18:57:07.708084 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:57:07.708051 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641376a9_6fb0_480e_bee1_f48c41b4ac89.slice/crio-3f6b956d451fd4f03fa4a85de515f43e156e6b2d5371814d215e67f6c2342370 WatchSource:0}: Error finding container 3f6b956d451fd4f03fa4a85de515f43e156e6b2d5371814d215e67f6c2342370: Status 404 returned error can't find the container with id 3f6b956d451fd4f03fa4a85de515f43e156e6b2d5371814d215e67f6c2342370 Nov 03 18:57:07.708657 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:07.708591 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87"] Nov 03 18:57:08.518753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:08.518633 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" event={"ID":"641376a9-6fb0-480e-bee1-f48c41b4ac89","Type":"ContainerStarted","Data":"f6f06a7b2569da48544ec59d1c4249da5c1a7cc6e6656b0436b5d838c96198a9"} Nov 03 18:57:08.518753 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:08.518717 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" event={"ID":"641376a9-6fb0-480e-bee1-f48c41b4ac89","Type":"ContainerStarted","Data":"3f6b956d451fd4f03fa4a85de515f43e156e6b2d5371814d215e67f6c2342370"} Nov 03 18:57:08.520499 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:08.520469 4948 generic.go:356] "Generic (PLEG): container finished" podID="4ada7914-3437-4c4e-b39f-04610941e91f" containerID="cec95e403f993f200356c0eb53de9b3faed438835642bc2f2f13fee76dc51926" exitCode=0 Nov 03 18:57:08.520623 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:08.520532 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" event={"ID":"4ada7914-3437-4c4e-b39f-04610941e91f","Type":"ContainerDied","Data":"cec95e403f993f200356c0eb53de9b3faed438835642bc2f2f13fee76dc51926"} Nov 03 18:57:08.640754 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:08.640685 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" podStartSLOduration=2.64067077 podStartE2EDuration="2.64067077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:08.639415266 +0000 UTC m=+1158.122411183" watchObservedRunningTime="2026-11-03 18:57:08.64067077 +0000 UTC m=+1158.123666775" Nov 03 18:57:09.549392 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:09.549352 4948 generic.go:356] "Generic (PLEG): container finished" podID="641376a9-6fb0-480e-bee1-f48c41b4ac89" containerID="f6f06a7b2569da48544ec59d1c4249da5c1a7cc6e6656b0436b5d838c96198a9" exitCode=0 Nov 03 18:57:09.550719 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:09.550663 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" event={"ID":"641376a9-6fb0-480e-bee1-f48c41b4ac89","Type":"ContainerDied","Data":"f6f06a7b2569da48544ec59d1c4249da5c1a7cc6e6656b0436b5d838c96198a9"} Nov 03 18:57:10.014725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.011955 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:10.147125 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.147078 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-openshift-ca\") pod \"4ada7914-3437-4c4e-b39f-04610941e91f\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " Nov 03 18:57:10.157745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.147652 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-xks-ca\") pod \"4ada7914-3437-4c4e-b39f-04610941e91f\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " Nov 03 18:57:10.157745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.147479 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-openshift-ca" pod "4ada7914-3437-4c4e-b39f-04610941e91f" (UID: "4ada7914-3437-4c4e-b39f-04610941e91f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:57:10.157745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.147715 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ada7914-3437-4c4e-b39f-04610941e91f-kube-api-access-4wfhv\" (UniqueName: \"kubernetes.io/projected/4ada7914-3437-4c4e-b39f-04610941e91f-kube-api-access-4wfhv\") pod \"4ada7914-3437-4c4e-b39f-04610941e91f\" (UID: \"4ada7914-3437-4c4e-b39f-04610941e91f\") " Nov 03 18:57:10.157745 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.148279 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:10.221010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.220951 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-xks-ca" pod "4ada7914-3437-4c4e-b39f-04610941e91f" (UID: "4ada7914-3437-4c4e-b39f-04610941e91f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:57:10.235123 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.232844 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ada7914-3437-4c4e-b39f-04610941e91f-kube-api-access-4wfhv" pod "4ada7914-3437-4c4e-b39f-04610941e91f" (UID: "4ada7914-3437-4c4e-b39f-04610941e91f"). InnerVolumeSpecName "kube-api-access-4wfhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:10.249458 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.248914 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ada7914-3437-4c4e-b39f-04610941e91f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:10.249458 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.248941 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wfhv\" (UniqueName: \"kubernetes.io/projected/4ada7914-3437-4c4e-b39f-04610941e91f-kube-api-access-4wfhv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:10.562383 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.562299 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" Nov 03 18:57:10.562827 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.562306 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9" event={"ID":"4ada7914-3437-4c4e-b39f-04610941e91f","Type":"ContainerDied","Data":"9987f1ce7341056ceb9f5fe48f8871bd6f3ae4056a52d94835635d437020c269"} Nov 03 18:57:10.562827 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.562443 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9987f1ce7341056ceb9f5fe48f8871bd6f3ae4056a52d94835635d437020c269" Nov 03 18:57:10.740598 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.740572 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:10.861299 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.861125 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-openshift-ca\") pod \"641376a9-6fb0-480e-bee1-f48c41b4ac89\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " Nov 03 18:57:10.861299 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.861178 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-xks-ca\") pod \"641376a9-6fb0-480e-bee1-f48c41b4ac89\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " Nov 03 18:57:10.861942 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.861892 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-openshift-ca" pod "641376a9-6fb0-480e-bee1-f48c41b4ac89" (UID: "641376a9-6fb0-480e-bee1-f48c41b4ac89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:57:10.869972 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.862249 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641376a9-6fb0-480e-bee1-f48c41b4ac89-kube-api-access-sswj8\" (UniqueName: \"kubernetes.io/projected/641376a9-6fb0-480e-bee1-f48c41b4ac89-kube-api-access-sswj8\") pod \"641376a9-6fb0-480e-bee1-f48c41b4ac89\" (UID: \"641376a9-6fb0-480e-bee1-f48c41b4ac89\") " Nov 03 18:57:10.869972 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.863091 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:10.869972 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.864853 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-xks-ca" pod "641376a9-6fb0-480e-bee1-f48c41b4ac89" (UID: "641376a9-6fb0-480e-bee1-f48c41b4ac89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:57:10.870218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.870064 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641376a9-6fb0-480e-bee1-f48c41b4ac89-kube-api-access-sswj8" pod "641376a9-6fb0-480e-bee1-f48c41b4ac89" (UID: "641376a9-6fb0-480e-bee1-f48c41b4ac89"). InnerVolumeSpecName "kube-api-access-sswj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:10.964144 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.964106 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641376a9-6fb0-480e-bee1-f48c41b4ac89-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:10.964144 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:10.964137 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sswj8\" (UniqueName: \"kubernetes.io/projected/641376a9-6fb0-480e-bee1-f48c41b4ac89-kube-api-access-sswj8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:11.576993 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:11.576960 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" Nov 03 18:57:11.577397 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:11.576952 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87" event={"ID":"641376a9-6fb0-480e-bee1-f48c41b4ac89","Type":"ContainerDied","Data":"3f6b956d451fd4f03fa4a85de515f43e156e6b2d5371814d215e67f6c2342370"} Nov 03 18:57:11.577397 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:11.577138 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6b956d451fd4f03fa4a85de515f43e156e6b2d5371814d215e67f6c2342370" Nov 03 18:57:14.164031 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.163991 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt"] Nov 03 18:57:14.164933 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.164906 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4ada7914-3437-4c4e-b39f-04610941e91f" containerName="revoke-keys" Nov 03 18:57:14.164933 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.164933 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ada7914-3437-4c4e-b39f-04610941e91f" containerName="revoke-keys" Nov 03 18:57:14.165049 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.164948 4948 cpu_manager.go:414] "containerMap: removing container" podUID="641376a9-6fb0-480e-bee1-f48c41b4ac89" containerName="revoke-keys" Nov 03 18:57:14.165049 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.164956 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641376a9-6fb0-480e-bee1-f48c41b4ac89" containerName="revoke-keys" Nov 03 18:57:14.165136 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.165123 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ada7914-3437-4c4e-b39f-04610941e91f" containerName="revoke-keys" Nov 03 18:57:14.165168 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.165141 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="641376a9-6fb0-480e-bee1-f48c41b4ac89" containerName="revoke-keys" Nov 03 18:57:14.169584 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.169560 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.205933 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.205902 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.206092 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.206030 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28l9s\" (UniqueName: \"kubernetes.io/projected/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-kube-api-access-28l9s\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.206219 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.206203 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.221622 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.221595 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt"] Nov 03 18:57:14.267909 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.267874 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:57:14.267909 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.267874 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:57:14.268104 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.267937 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:57:14.273843 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.273818 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:57:14.306988 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.306964 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.307127 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.307003 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.307166 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.307151 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28l9s\" (UniqueName: \"kubernetes.io/projected/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-kube-api-access-28l9s\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.307693 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.307671 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.309377 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.309356 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.369925 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.369896 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28l9s\" (UniqueName: \"kubernetes.io/projected/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-kube-api-access-28l9s\") pod \"maas-api-revoke-sub-3a65e4d8-fcjbt\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.484195 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.484161 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:14.863568 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:57:14.863534 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a318f7b_c0b6_4f1e_8a4e_cbf6287ecb6e.slice/crio-62ad753cc704e03ffc11c37c62605298f55139d177da1e543b3500da4659b41c WatchSource:0}: Error finding container 62ad753cc704e03ffc11c37c62605298f55139d177da1e543b3500da4659b41c: Status 404 returned error can't find the container with id 62ad753cc704e03ffc11c37c62605298f55139d177da1e543b3500da4659b41c Nov 03 18:57:14.864424 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:14.864393 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt"] Nov 03 18:57:15.616217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:15.616187 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" event={"ID":"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e","Type":"ContainerStarted","Data":"f4f16f6179c4eb8f31eca8d251831c86187f9f6bd5d879031483e5e49d5af4fa"} Nov 03 18:57:15.616588 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:15.616237 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" event={"ID":"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e","Type":"ContainerStarted","Data":"62ad753cc704e03ffc11c37c62605298f55139d177da1e543b3500da4659b41c"} Nov 03 18:57:15.734882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:15.734833 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" podStartSLOduration=1.7348205540000001 podStartE2EDuration="1.734820554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:15.733931632 +0000 UTC m=+1165.216927545" watchObservedRunningTime="2026-11-03 18:57:15.734820554 +0000 UTC m=+1165.217816508" Nov 03 18:57:16.625584 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:16.625546 4948 generic.go:356] "Generic (PLEG): container finished" podID="7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" containerID="f4f16f6179c4eb8f31eca8d251831c86187f9f6bd5d879031483e5e49d5af4fa" exitCode=0 Nov 03 18:57:16.626124 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:16.625642 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" event={"ID":"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e","Type":"ContainerDied","Data":"f4f16f6179c4eb8f31eca8d251831c86187f9f6bd5d879031483e5e49d5af4fa"} Nov 03 18:57:17.788362 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.788333 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:17.845221 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.845191 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-kube-api-access-28l9s\" (UniqueName: \"kubernetes.io/projected/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-kube-api-access-28l9s\") pod \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " Nov 03 18:57:17.845370 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.845250 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-openshift-ca\") pod \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " Nov 03 18:57:17.845370 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.845355 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-xks-ca\") pod \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\" (UID: \"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e\") " Nov 03 18:57:17.845658 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.845626 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-openshift-ca" pod "7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" (UID: "7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:57:17.845911 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.845890 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:17.847432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.847410 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-xks-ca" pod "7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" (UID: "7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:57:17.847432 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.847418 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-kube-api-access-28l9s" pod "7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" (UID: "7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e"). InnerVolumeSpecName "kube-api-access-28l9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:17.946800 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.946764 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28l9s\" (UniqueName: \"kubernetes.io/projected/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-kube-api-access-28l9s\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:17.946800 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:17.946795 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:18.642287 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:18.642258 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" Nov 03 18:57:18.642487 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:18.642293 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt" event={"ID":"7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e","Type":"ContainerDied","Data":"62ad753cc704e03ffc11c37c62605298f55139d177da1e543b3500da4659b41c"} Nov 03 18:57:18.642487 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:18.642340 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62ad753cc704e03ffc11c37c62605298f55139d177da1e543b3500da4659b41c" Nov 03 18:57:22.250496 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.250449 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5"] Nov 03 18:57:22.251256 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.251233 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" containerName="revoke-keys" Nov 03 18:57:22.251361 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.251257 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" containerName="revoke-keys" Nov 03 18:57:22.251419 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.251392 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" containerName="revoke-keys" Nov 03 18:57:22.254590 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.254566 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.309945 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.309912 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5"] Nov 03 18:57:22.355562 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.355529 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:57:22.355747 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.355603 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:57:22.355747 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.355670 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:57:22.358855 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.358834 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:57:22.400055 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.400025 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.400238 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.400216 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sknz\" (UniqueName: \"kubernetes.io/projected/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-kube-api-access-2sknz\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.400356 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.400333 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.501054 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.500974 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.501054 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.501028 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.501256 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.501140 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sknz\" (UniqueName: \"kubernetes.io/projected/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-kube-api-access-2sknz\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.501804 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.501777 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.503441 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.503417 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.563612 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.563583 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sknz\" (UniqueName: \"kubernetes.io/projected/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-kube-api-access-2sknz\") pod \"maas-api-revoke-sub-22aee09a-t4lz5\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.569402 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.569378 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:22.961205 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:57:22.961177 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56b4d7b_f600_4058_8ab8_7ede8bf2357a.slice/crio-502d3bcd01c46776469361fa20e5060c6e46358e204739a452fc14079e24601c WatchSource:0}: Error finding container 502d3bcd01c46776469361fa20e5060c6e46358e204739a452fc14079e24601c: Status 404 returned error can't find the container with id 502d3bcd01c46776469361fa20e5060c6e46358e204739a452fc14079e24601c Nov 03 18:57:22.961738 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:22.961715 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5"] Nov 03 18:57:23.686390 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:23.686353 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" event={"ID":"c56b4d7b-f600-4058-8ab8-7ede8bf2357a","Type":"ContainerStarted","Data":"b1aa4321ae5130682a024410ce73ed0aa8db211a9e9fd2118a0c74d95aae7f48"} Nov 03 18:57:23.686788 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:23.686441 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" event={"ID":"c56b4d7b-f600-4058-8ab8-7ede8bf2357a","Type":"ContainerStarted","Data":"502d3bcd01c46776469361fa20e5060c6e46358e204739a452fc14079e24601c"} Nov 03 18:57:23.808770 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:23.808690 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" podStartSLOduration=1.80867382 podStartE2EDuration="1.80867382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:23.804354628 +0000 UTC m=+1173.287350566" watchObservedRunningTime="2026-11-03 18:57:23.80867382 +0000 UTC m=+1173.291669734" Nov 03 18:57:24.698468 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:24.698428 4948 generic.go:356] "Generic (PLEG): container finished" podID="c56b4d7b-f600-4058-8ab8-7ede8bf2357a" containerID="b1aa4321ae5130682a024410ce73ed0aa8db211a9e9fd2118a0c74d95aae7f48" exitCode=0 Nov 03 18:57:24.698924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:24.698516 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" event={"ID":"c56b4d7b-f600-4058-8ab8-7ede8bf2357a","Type":"ContainerDied","Data":"b1aa4321ae5130682a024410ce73ed0aa8db211a9e9fd2118a0c74d95aae7f48"} Nov 03 18:57:25.859027 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.859004 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:25.939032 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.939001 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-openshift-ca\") pod \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " Nov 03 18:57:25.939218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.939144 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-kube-api-access-2sknz\" (UniqueName: \"kubernetes.io/projected/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-kube-api-access-2sknz\") pod \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " Nov 03 18:57:25.939218 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.939181 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-xks-ca\") pod \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\" (UID: \"c56b4d7b-f600-4058-8ab8-7ede8bf2357a\") " Nov 03 18:57:25.939349 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.939313 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-openshift-ca" pod "c56b4d7b-f600-4058-8ab8-7ede8bf2357a" (UID: "c56b4d7b-f600-4058-8ab8-7ede8bf2357a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:57:25.939528 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.939509 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:25.941197 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.941165 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-xks-ca" pod "c56b4d7b-f600-4058-8ab8-7ede8bf2357a" (UID: "c56b4d7b-f600-4058-8ab8-7ede8bf2357a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:57:25.941307 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:25.941243 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-kube-api-access-2sknz" pod "c56b4d7b-f600-4058-8ab8-7ede8bf2357a" (UID: "c56b4d7b-f600-4058-8ab8-7ede8bf2357a"). InnerVolumeSpecName "kube-api-access-2sknz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:26.040541 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:26.040473 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:26.040541 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:26.040498 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sknz\" (UniqueName: \"kubernetes.io/projected/c56b4d7b-f600-4058-8ab8-7ede8bf2357a-kube-api-access-2sknz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:26.715694 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:26.715662 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" Nov 03 18:57:26.715694 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:26.715688 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5" event={"ID":"c56b4d7b-f600-4058-8ab8-7ede8bf2357a","Type":"ContainerDied","Data":"502d3bcd01c46776469361fa20e5060c6e46358e204739a452fc14079e24601c"} Nov 03 18:57:26.715924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:26.715751 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="502d3bcd01c46776469361fa20e5060c6e46358e204739a452fc14079e24601c" Nov 03 18:57:32.703563 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.703528 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz"] Nov 03 18:57:32.704113 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.704099 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c56b4d7b-f600-4058-8ab8-7ede8bf2357a" containerName="revoke-keys" Nov 03 18:57:32.704157 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.704114 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56b4d7b-f600-4058-8ab8-7ede8bf2357a" containerName="revoke-keys" Nov 03 18:57:32.704231 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.704222 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56b4d7b-f600-4058-8ab8-7ede8bf2357a" containerName="revoke-keys" Nov 03 18:57:32.709078 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.709063 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.768134 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.768074 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz"] Nov 03 18:57:32.799748 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:57:32.799685 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:57:32.813151 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.813124 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-355e1a-openshift-default-dockercfg-7nkhf\"" Nov 03 18:57:32.827379 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827356 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827508 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827398 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827508 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827469 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827513 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827615 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827541 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kfs\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-kube-api-access-79kfs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827690 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827663 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827756 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827691 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827878 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827857 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.827925 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.827914 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.928612 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.928579 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.928820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.928623 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.928820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.928684 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.928820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.928747 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.928820 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.928770 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929161 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929021 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929161 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929080 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929161 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929093 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kfs\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-kube-api-access-79kfs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929161 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929109 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929470 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929166 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929470 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929170 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929470 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929191 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929470 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929447 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.929786 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.929768 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.931252 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.931227 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.931347 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.931266 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.991549 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.991463 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:32.991839 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:32.991629 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kfs\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-kube-api-access-79kfs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:33.024169 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.024141 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:33.465870 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:57:33.465824 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3829a1_7c3f_40d9_8b16_3f0d302e55a8.slice/crio-97b176e649b6bed327ab49002b57c374f9b409873b10f56108a462d82496dfcb WatchSource:0}: Error finding container 97b176e649b6bed327ab49002b57c374f9b409873b10f56108a462d82496dfcb: Status 404 returned error can't find the container with id 97b176e649b6bed327ab49002b57c374f9b409873b10f56108a462d82496dfcb Nov 03 18:57:33.468477 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.468439 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:57:33.468578 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.468529 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:57:33.468619 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.468580 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:57:33.468656 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.468604 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz"] Nov 03 18:57:33.653877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.653841 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck"] Nov 03 18:57:33.671565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.671497 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.742866 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.742793 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck"] Nov 03 18:57:33.786772 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.786733 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" event={"ID":"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8","Type":"ContainerStarted","Data":"3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa"} Nov 03 18:57:33.786963 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.786803 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" event={"ID":"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8","Type":"ContainerStarted","Data":"97b176e649b6bed327ab49002b57c374f9b409873b10f56108a462d82496dfcb"} Nov 03 18:57:33.840246 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840209 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.840417 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840273 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrj7\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-kube-api-access-9vrj7\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.840507 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840485 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.840565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840527 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.840669 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840645 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.840767 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840723 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.840889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840868 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.840963 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840947 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.841035 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.840975 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.919993 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.919937 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podStartSLOduration=1.919921486 podStartE2EDuration="1.919921486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:33.914283174 +0000 UTC m=+1183.397279087" watchObservedRunningTime="2026-11-03 18:57:33.919921486 +0000 UTC m=+1183.402917399" Nov 03 18:57:33.941889 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.941852 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942049 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.941911 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942515 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942489 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942609 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942545 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942658 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.941975 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942730 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942669 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942724 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942771 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942797 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942794 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942966 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942848 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.942966 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.942881 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrj7\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-kube-api-access-9vrj7\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.943078 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.943013 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.943676 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.943653 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.943796 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.943761 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.945807 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.945789 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:33.946186 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:33.946166 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:34.004236 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.004155 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:34.005561 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.005527 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrj7\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-kube-api-access-9vrj7\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:34.026717 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.026081 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:34.033521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.032439 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:57:34.033521 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.032529 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:57:34.289503 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.289409 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:34.784901 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.784867 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:57:34.785315 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.784947 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:57:34.785315 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.784994 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:57:34.786611 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.786585 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck"] Nov 03 18:57:34.800610 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:34.800584 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" event={"ID":"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19","Type":"ContainerStarted","Data":"0a6f9d8757bff660d5510a9e9585cc7b0176ef53175b3937f120716b6146a570"} Nov 03 18:57:35.024575 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:35.024497 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:57:35.024575 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:35.024551 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:57:35.813391 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:35.813350 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" event={"ID":"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19","Type":"ContainerStarted","Data":"6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff"} Nov 03 18:57:36.024990 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:36.024960 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:57:36.025134 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:36.025007 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:57:36.293724 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:36.292919 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:36.293724 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:36.293060 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 18:57:36.293724 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:36.293110 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 18:57:37.027140 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:37.027095 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:57:37.027645 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:37.027176 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:57:37.295236 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:37.295133 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 18:57:37.295495 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:37.295463 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 18:57:38.025426 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:38.025388 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:57:38.025591 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:38.025441 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:57:38.295784 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:38.295588 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 18:57:38.295784 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:38.295653 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 18:57:39.025485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:39.025452 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:57:39.025671 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:39.025503 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:57:39.296982 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:39.296889 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 18:57:39.297555 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:39.297526 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 18:57:40.024762 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:40.024721 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:57:40.024957 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:40.024791 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:57:40.297790 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:40.293066 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 18:57:40.297790 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:40.293132 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 18:57:41.029605 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:41.029574 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:41.029967 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:41.029940 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:41.030814 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:41.030795 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 18:57:41.256875 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:41.256822 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podStartSLOduration=8.256807738 podStartE2EDuration="8.256807738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:36.006377467 +0000 UTC m=+1185.489373398" watchObservedRunningTime="2026-11-03 18:57:41.256807738 +0000 UTC m=+1190.739803651" Nov 03 18:57:41.290207 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:41.290117 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 18:57:41.290207 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:41.290177 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 18:57:42.295131 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:42.295101 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:42.295582 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:42.295561 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:42.296373 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:42.296356 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 18:57:50.909851 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:50.909821 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:57:50.915554 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:50.915529 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 18:57:50.917338 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:50.917259 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:57:50.922850 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:57:50.922831 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 18:58:02.793227 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:58:02.793144 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:58:22.192977 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.192940 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc"] Nov 03 18:58:22.197374 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.197352 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.228061 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.228013 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tls-certs\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.228219 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.228173 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-model-cache\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.228328 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.228302 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-home\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.228378 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.228362 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-dshm\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.228490 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.228473 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tmp-dir\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.228562 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.228544 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtcb\" (UniqueName: \"kubernetes.io/projected/46553f9b-0d60-45c3-aabe-2a608c8df7fc-kube-api-access-7gtcb\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.251847 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.251814 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc"] Nov 03 18:58:22.297804 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.297770 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"kube-root-ca.crt\"" Nov 03 18:58:22.297804 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.297777 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"openshift-service-ca.crt\"" Nov 03 18:58:22.298012 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.297819 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"default-dockercfg-h7sxm\"" Nov 03 18:58:22.298012 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.297830 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"distinct-w5-3a7056-kserve-self-signed-certs\"" Nov 03 18:58:22.335658 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.335610 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-home\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.340794 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.336512 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-dshm\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.341130 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.341087 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tmp-dir\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.343035 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.343005 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtcb\" (UniqueName: \"kubernetes.io/projected/46553f9b-0d60-45c3-aabe-2a608c8df7fc-kube-api-access-7gtcb\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.343276 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.343255 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tls-certs\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.350039 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.343507 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-model-cache\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.350322 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.336437 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-home\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.350858 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.350757 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tmp-dir\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.351008 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.350797 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-model-cache\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.353818 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.353793 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tls-certs\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.357373 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.356996 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-dshm\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.411525 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.411496 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtcb\" (UniqueName: \"kubernetes.io/projected/46553f9b-0d60-45c3-aabe-2a608c8df7fc-kube-api-access-7gtcb\") pod \"distinct-w5-3a7056-kserve-6dd597b6c6-998bc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.511884 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.511788 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:22.954071 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:58:22.954041 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46553f9b_0d60_45c3_aabe_2a608c8df7fc.slice/crio-8084d4e6296f74780d4faf7e4dd439d2fd75cb3adea67b5f6342648a68ec4ea9 WatchSource:0}: Error finding container 8084d4e6296f74780d4faf7e4dd439d2fd75cb3adea67b5f6342648a68ec4ea9: Status 404 returned error can't find the container with id 8084d4e6296f74780d4faf7e4dd439d2fd75cb3adea67b5f6342648a68ec4ea9 Nov 03 18:58:22.954850 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:22.954690 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc"] Nov 03 18:58:23.226357 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:23.226269 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" event={"ID":"46553f9b-0d60-45c3-aabe-2a608c8df7fc","Type":"ContainerStarted","Data":"ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63"} Nov 03 18:58:23.226357 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:23.226316 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" event={"ID":"46553f9b-0d60-45c3-aabe-2a608c8df7fc","Type":"ContainerStarted","Data":"8084d4e6296f74780d4faf7e4dd439d2fd75cb3adea67b5f6342648a68ec4ea9"} Nov 03 18:58:23.348217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:23.348164 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podStartSLOduration=1.348148044 podStartE2EDuration="1.348148044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:23.344471972 +0000 UTC m=+1232.827467886" watchObservedRunningTime="2026-11-03 18:58:23.348148044 +0000 UTC m=+1232.831143959" Nov 03 18:58:23.512068 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:23.511989 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:32.512883 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:32.512849 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:32.525646 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:32.525617 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:32.792935 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:58:32.792848 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:58:33.314727 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:33.314670 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 18:58:49.715414 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.715380 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw"] Nov 03 18:58:49.719931 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.719908 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.775986 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.775947 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw"] Nov 03 18:58:49.784725 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.784679 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-home\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.784987 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.784957 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-model-cache\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.785087 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.785027 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg8d2\" (UniqueName: \"kubernetes.io/projected/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-kube-api-access-xg8d2\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.785187 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.785161 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tls-certs\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.785321 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.785296 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-dshm\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.785395 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.785379 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tmp-dir\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.818890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.818862 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"distinct-2-w3-a4cacd-kserve-self-signed-certs\"" Nov 03 18:58:49.886600 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.886554 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-home\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.886921 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.886885 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-model-cache\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.887123 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.887091 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg8d2\" (UniqueName: \"kubernetes.io/projected/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-kube-api-access-xg8d2\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.887317 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.887302 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tls-certs\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.887511 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.887497 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-dshm\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.887654 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.887640 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tmp-dir\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.888522 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.888492 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tmp-dir\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.892634 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.892608 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-home\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.893206 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.893184 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-model-cache\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.902857 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.902827 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-dshm\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.903647 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.903612 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tls-certs\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:49.957906 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:49.957876 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg8d2\" (UniqueName: \"kubernetes.io/projected/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-kube-api-access-xg8d2\") pod \"distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:50.035125 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:50.035044 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:58:50.499867 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:58:50.499827 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f6d0ca0_ce58_4983_bc97_8222e54b8c1c.slice/crio-599ffcbc66c987bfb12dec250e9a7de9f983f1e4f7a9b8478279f24319f438b1 WatchSource:0}: Error finding container 599ffcbc66c987bfb12dec250e9a7de9f983f1e4f7a9b8478279f24319f438b1: Status 404 returned error can't find the container with id 599ffcbc66c987bfb12dec250e9a7de9f983f1e4f7a9b8478279f24319f438b1 Nov 03 18:58:50.500924 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:50.500896 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw"] Nov 03 18:58:51.467967 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:51.467925 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" event={"ID":"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c","Type":"ContainerStarted","Data":"1617fda5cdf9e5194bd8a78ad342305816f0e1fcee3824da849e7f3622c9fa44"} Nov 03 18:58:51.468485 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:51.467980 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" event={"ID":"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c","Type":"ContainerStarted","Data":"599ffcbc66c987bfb12dec250e9a7de9f983f1e4f7a9b8478279f24319f438b1"} Nov 03 18:58:51.587541 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:51.587469 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podStartSLOduration=2.587449195 podStartE2EDuration="2.587449195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:51.586557223 +0000 UTC m=+1261.069553139" watchObservedRunningTime="2026-11-03 18:58:51.587449195 +0000 UTC m=+1261.070445109" Nov 03 18:58:52.035648 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:58:52.035615 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:59:00.035578 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:00.035536 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:59:00.049130 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:00.049106 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:59:00.571179 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:00.571148 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 18:59:02.793835 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:59:02.793797 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:59:21.376480 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.376428 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg"] Nov 03 18:59:21.382025 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.382004 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.432426 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.432385 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg"] Nov 03 18:59:21.478777 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.478742 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:59:21.478777 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.478749 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:59:21.478777 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.478749 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:59:21.479621 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.479598 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.479739 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.479654 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.479800 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.479750 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxf8g\" (UniqueName: \"kubernetes.io/projected/44a9dc4c-5b14-4d02-8605-3d7501316903-kube-api-access-zxf8g\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.491877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.491853 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:59:21.580897 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.580859 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.581063 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.580919 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.581063 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.580967 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxf8g\" (UniqueName: \"kubernetes.io/projected/44a9dc4c-5b14-4d02-8605-3d7501316903-kube-api-access-zxf8g\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.581694 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.581664 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.583348 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.583327 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.643565 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.643497 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxf8g\" (UniqueName: \"kubernetes.io/projected/44a9dc4c-5b14-4d02-8605-3d7501316903-kube-api-access-zxf8g\") pod \"maas-api-revoke-sub-47f011ae-zcwtg\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:21.698647 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:21.698611 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:22.080207 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:22.080179 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg"] Nov 03 18:59:22.745070 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:22.745038 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" event={"ID":"44a9dc4c-5b14-4d02-8605-3d7501316903","Type":"ContainerStarted","Data":"1382db2ade5075e7dea829ca3e62763879913af7e323c0b7f6d09d39344843bb"} Nov 03 18:59:22.745465 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:22.745087 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" event={"ID":"44a9dc4c-5b14-4d02-8605-3d7501316903","Type":"ContainerStarted","Data":"21c37c49de9a7062d1f8b89d0078d613a300f3e4844993f4dc1243ab12835829"} Nov 03 18:59:22.865010 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:22.864958 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" podStartSLOduration=1.864940437 podStartE2EDuration="1.864940437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:22.864232823 +0000 UTC m=+1292.347228738" watchObservedRunningTime="2026-11-03 18:59:22.864940437 +0000 UTC m=+1292.347936352" Nov 03 18:59:23.754747 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:23.754680 4948 generic.go:356] "Generic (PLEG): container finished" podID="44a9dc4c-5b14-4d02-8605-3d7501316903" containerID="1382db2ade5075e7dea829ca3e62763879913af7e323c0b7f6d09d39344843bb" exitCode=0 Nov 03 18:59:23.755248 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:23.754773 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" event={"ID":"44a9dc4c-5b14-4d02-8605-3d7501316903","Type":"ContainerDied","Data":"1382db2ade5075e7dea829ca3e62763879913af7e323c0b7f6d09d39344843bb"} Nov 03 18:59:24.943540 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:24.943516 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:25.023733 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.023648 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-xks-ca\") pod \"44a9dc4c-5b14-4d02-8605-3d7501316903\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " Nov 03 18:59:25.023877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.023742 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a9dc4c-5b14-4d02-8605-3d7501316903-kube-api-access-zxf8g\" (UniqueName: \"kubernetes.io/projected/44a9dc4c-5b14-4d02-8605-3d7501316903-kube-api-access-zxf8g\") pod \"44a9dc4c-5b14-4d02-8605-3d7501316903\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " Nov 03 18:59:25.023877 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.023785 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-openshift-ca\") pod \"44a9dc4c-5b14-4d02-8605-3d7501316903\" (UID: \"44a9dc4c-5b14-4d02-8605-3d7501316903\") " Nov 03 18:59:25.024199 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.024168 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-openshift-ca" pod "44a9dc4c-5b14-4d02-8605-3d7501316903" (UID: "44a9dc4c-5b14-4d02-8605-3d7501316903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:59:25.025663 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.025634 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-xks-ca" pod "44a9dc4c-5b14-4d02-8605-3d7501316903" (UID: "44a9dc4c-5b14-4d02-8605-3d7501316903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:59:25.025776 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.025676 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a9dc4c-5b14-4d02-8605-3d7501316903-kube-api-access-zxf8g" pod "44a9dc4c-5b14-4d02-8605-3d7501316903" (UID: "44a9dc4c-5b14-4d02-8605-3d7501316903"). InnerVolumeSpecName "kube-api-access-zxf8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:59:25.124776 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.124740 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxf8g\" (UniqueName: \"kubernetes.io/projected/44a9dc4c-5b14-4d02-8605-3d7501316903-kube-api-access-zxf8g\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:59:25.124776 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.124771 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:59:25.124776 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.124783 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a9dc4c-5b14-4d02-8605-3d7501316903-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:59:25.772822 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.772788 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" event={"ID":"44a9dc4c-5b14-4d02-8605-3d7501316903","Type":"ContainerDied","Data":"21c37c49de9a7062d1f8b89d0078d613a300f3e4844993f4dc1243ab12835829"} Nov 03 18:59:25.772998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.772838 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c37c49de9a7062d1f8b89d0078d613a300f3e4844993f4dc1243ab12835829" Nov 03 18:59:25.772998 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:25.772801 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg" Nov 03 18:59:32.797445 ip-10-0-128-95 kubenswrapper[4948]: E1103 18:59:32.797345 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 18:59:33.315011 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.314980 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2"] Nov 03 18:59:33.315572 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.315557 4948 cpu_manager.go:414] "containerMap: removing container" podUID="44a9dc4c-5b14-4d02-8605-3d7501316903" containerName="revoke-keys" Nov 03 18:59:33.315621 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.315573 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a9dc4c-5b14-4d02-8605-3d7501316903" containerName="revoke-keys" Nov 03 18:59:33.315689 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.315680 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a9dc4c-5b14-4d02-8605-3d7501316903" containerName="revoke-keys" Nov 03 18:59:33.319197 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.319182 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.428958 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.428915 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2"] Nov 03 18:59:33.433576 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.433548 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27gr\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-kube-api-access-k27gr\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.433717 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.433591 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.433717 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.433632 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.433858 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.433829 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.433914 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.433884 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.433985 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.433962 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a69ca-cec8-4640-82cd-04b021bea8e5-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.434114 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.434086 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.434155 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.434137 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.434318 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.434302 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.535726 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535671 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27gr\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-kube-api-access-k27gr\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.535890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535748 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.535890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535785 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.535890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535826 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.535890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535850 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.535890 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535868 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a69ca-cec8-4640-82cd-04b021bea8e5-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536163 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535919 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536163 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.535944 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536163 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.536072 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536473 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.536446 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536585 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.536560 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.536596 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536815 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.536792 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a69ca-cec8-4640-82cd-04b021bea8e5-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.536858 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.536812 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.538107 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.538083 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.538389 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.538372 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.598053 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.597987 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.598053 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.598038 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27gr\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-kube-api-access-k27gr\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:33.632984 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:33.632959 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:34.072667 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:59:34.072636 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182a69ca_cec8_4640_82cd_04b021bea8e5.slice/crio-d74962b518c1f205abd08d96548380ff45735a408f4aef8a52f010fd9ffea56e WatchSource:0}: Error finding container d74962b518c1f205abd08d96548380ff45735a408f4aef8a52f010fd9ffea56e: Status 404 returned error can't find the container with id d74962b518c1f205abd08d96548380ff45735a408f4aef8a52f010fd9ffea56e Nov 03 18:59:34.073120 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:34.073077 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2"] Nov 03 18:59:34.075305 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:34.075262 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:59:34.075378 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:34.075359 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:59:34.075423 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:34.075407 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:59:34.848156 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:34.848123 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" event={"ID":"182a69ca-cec8-4640-82cd-04b021bea8e5","Type":"ContainerStarted","Data":"b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640"} Nov 03 18:59:34.848335 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:34.848176 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" event={"ID":"182a69ca-cec8-4640-82cd-04b021bea8e5","Type":"ContainerStarted","Data":"d74962b518c1f205abd08d96548380ff45735a408f4aef8a52f010fd9ffea56e"} Nov 03 18:59:35.007530 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:35.007473 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podStartSLOduration=2.007457427 podStartE2EDuration="2.007457427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:35.00251433 +0000 UTC m=+1304.485510243" watchObservedRunningTime="2026-11-03 18:59:35.007457427 +0000 UTC m=+1304.490453340" Nov 03 18:59:35.644477 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:35.644246 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:35.644477 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:35.644411 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:59:35.644477 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:35.644459 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:59:36.633422 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:36.633380 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:59:36.633588 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:36.633448 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:59:37.633415 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:37.633371 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:59:37.633844 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:37.633450 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:59:38.633667 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:38.633630 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:59:38.634139 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:38.633727 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:59:39.633445 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:39.633404 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:59:39.633626 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:39.633475 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:59:40.633823 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:40.633784 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:59:40.634217 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:40.633855 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:59:41.638392 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:41.638364 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:41.638876 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:41.638810 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:41.639538 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:41.639518 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 18:59:52.057929 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.057894 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t"] Nov 03 18:59:52.062292 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.062270 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.124730 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.124674 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t"] Nov 03 18:59:52.164956 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.164923 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:59:52.165122 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.164986 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:59:52.165122 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.164990 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:59:52.169408 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.169390 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 18:59:52.192541 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.192512 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.192673 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.192656 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmdq\" (UniqueName: \"kubernetes.io/projected/cbcfc66a-658f-43f5-99b2-2600b76d3806-kube-api-access-bdmdq\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.192842 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.192821 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.294258 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.294225 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.294419 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.294322 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmdq\" (UniqueName: \"kubernetes.io/projected/cbcfc66a-658f-43f5-99b2-2600b76d3806-kube-api-access-bdmdq\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.294419 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.294395 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.295026 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.295000 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.296749 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.296718 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.356443 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.356371 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmdq\" (UniqueName: \"kubernetes.io/projected/cbcfc66a-658f-43f5-99b2-2600b76d3806-kube-api-access-bdmdq\") pod \"maas-api-revoke-sub-0f810b24-tv79t\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.380267 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.380241 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:52.761455 ip-10-0-128-95 kubenswrapper[4948]: W1103 18:59:52.761429 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcfc66a_658f_43f5_99b2_2600b76d3806.slice/crio-b751136bf21bf8a83b175b45c48662f6ee8262ea8d444bc06a410306c5efc5ba WatchSource:0}: Error finding container b751136bf21bf8a83b175b45c48662f6ee8262ea8d444bc06a410306c5efc5ba: Status 404 returned error can't find the container with id b751136bf21bf8a83b175b45c48662f6ee8262ea8d444bc06a410306c5efc5ba Nov 03 18:59:52.762257 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:52.762236 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t"] Nov 03 18:59:53.054316 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:53.054280 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" event={"ID":"cbcfc66a-658f-43f5-99b2-2600b76d3806","Type":"ContainerStarted","Data":"b751136bf21bf8a83b175b45c48662f6ee8262ea8d444bc06a410306c5efc5ba"} Nov 03 18:59:54.063994 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:54.063959 4948 generic.go:356] "Generic (PLEG): container finished" podID="cbcfc66a-658f-43f5-99b2-2600b76d3806" containerID="a5ca5f73f5a48d9945176e7f5087e73dbfb96d4cd9838d88e746b14870aaa58b" exitCode=0 Nov 03 18:59:54.064386 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:54.064039 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" event={"ID":"cbcfc66a-658f-43f5-99b2-2600b76d3806","Type":"ContainerDied","Data":"a5ca5f73f5a48d9945176e7f5087e73dbfb96d4cd9838d88e746b14870aaa58b"} Nov 03 18:59:55.232311 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.232285 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:55.329547 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.329469 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcfc66a-658f-43f5-99b2-2600b76d3806-kube-api-access-bdmdq\" (UniqueName: \"kubernetes.io/projected/cbcfc66a-658f-43f5-99b2-2600b76d3806-kube-api-access-bdmdq\") pod \"cbcfc66a-658f-43f5-99b2-2600b76d3806\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " Nov 03 18:59:55.329723 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.329548 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-xks-ca\") pod \"cbcfc66a-658f-43f5-99b2-2600b76d3806\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " Nov 03 18:59:55.329723 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.329573 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-openshift-ca\") pod \"cbcfc66a-658f-43f5-99b2-2600b76d3806\" (UID: \"cbcfc66a-658f-43f5-99b2-2600b76d3806\") " Nov 03 18:59:55.330093 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.330064 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-openshift-ca" pod "cbcfc66a-658f-43f5-99b2-2600b76d3806" (UID: "cbcfc66a-658f-43f5-99b2-2600b76d3806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:59:55.331633 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.331603 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-xks-ca" pod "cbcfc66a-658f-43f5-99b2-2600b76d3806" (UID: "cbcfc66a-658f-43f5-99b2-2600b76d3806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:59:55.331785 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.331737 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcfc66a-658f-43f5-99b2-2600b76d3806-kube-api-access-bdmdq" pod "cbcfc66a-658f-43f5-99b2-2600b76d3806" (UID: "cbcfc66a-658f-43f5-99b2-2600b76d3806"). InnerVolumeSpecName "kube-api-access-bdmdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:59:55.430882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.430851 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:59:55.430882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.430879 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcfc66a-658f-43f5-99b2-2600b76d3806-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:59:55.430882 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:55.430890 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmdq\" (UniqueName: \"kubernetes.io/projected/cbcfc66a-658f-43f5-99b2-2600b76d3806-kube-api-access-bdmdq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:59:56.080985 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:56.080952 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" Nov 03 18:59:56.080985 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:56.080975 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t" event={"ID":"cbcfc66a-658f-43f5-99b2-2600b76d3806","Type":"ContainerDied","Data":"b751136bf21bf8a83b175b45c48662f6ee8262ea8d444bc06a410306c5efc5ba"} Nov 03 18:59:56.081196 ip-10-0-128-95 kubenswrapper[4948]: I1103 18:59:56.081025 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b751136bf21bf8a83b175b45c48662f6ee8262ea8d444bc06a410306c5efc5ba" Nov 03 19:00:00.175760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.175719 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms"] Nov 03 19:00:00.176629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.176607 4948 cpu_manager.go:414] "containerMap: removing container" podUID="cbcfc66a-658f-43f5-99b2-2600b76d3806" containerName="revoke-keys" Nov 03 19:00:00.176764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.176632 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcfc66a-658f-43f5-99b2-2600b76d3806" containerName="revoke-keys" Nov 03 19:00:00.176869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.176856 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcfc66a-658f-43f5-99b2-2600b76d3806" containerName="revoke-keys" Nov 03 19:00:00.180622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.180601 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl"] Nov 03 19:00:00.180793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.180771 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.184937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.184917 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.186502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.185901 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g"] Nov 03 19:00:00.190123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.190106 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.226272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.226245 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b"] Nov 03 19:00:00.230753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.230735 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.230848 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.230834 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc"] Nov 03 19:00:00.234566 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.234548 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:00:00.234650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.234578 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:00:00.234746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.234716 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:00:00.235013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.234996 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.284007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.283967 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms"] Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.284909 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285001 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7588e6c6-b872-4193-8de8-990012b2e689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285046 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dff7e1-48b6-44b9-a59e-5f409d535387-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-b4sms\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285106 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59xz\" (UniqueName: \"kubernetes.io/projected/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-kube-api-access-f59xz\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285131 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rj8\" (UniqueName: \"kubernetes.io/projected/01dff7e1-48b6-44b9-a59e-5f409d535387-kube-api-access-p8rj8\") pod \"maas-api-key-cleanup-29895540-b4sms\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285193 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285229 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkcq4\" (UniqueName: \"kubernetes.io/projected/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-kube-api-access-pkcq4\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285254 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285295 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvkd\" (UniqueName: \"kubernetes.io/projected/7588e6c6-b872-4193-8de8-990012b2e689-kube-api-access-csvkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.285738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.285371 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhkb\" (UniqueName: \"kubernetes.io/projected/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-kube-api-access-wxhkb\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.287819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.287785 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl"] Nov 03 19:00:00.292956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.292930 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b"] Nov 03 19:00:00.298417 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.297388 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g"] Nov 03 19:00:00.321656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.321619 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc"] Nov 03 19:00:00.386809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.386764 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvkd\" (UniqueName: \"kubernetes.io/projected/7588e6c6-b872-4193-8de8-990012b2e689-kube-api-access-csvkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.387011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.386862 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhkb\" (UniqueName: \"kubernetes.io/projected/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-kube-api-access-wxhkb\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.387011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.386925 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.387011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387005 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7588e6c6-b872-4193-8de8-990012b2e689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.387183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387045 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dff7e1-48b6-44b9-a59e-5f409d535387-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-b4sms\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.387183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387137 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59xz\" (UniqueName: \"kubernetes.io/projected/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-kube-api-access-f59xz\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.387183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387164 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rj8\" (UniqueName: \"kubernetes.io/projected/01dff7e1-48b6-44b9-a59e-5f409d535387-kube-api-access-p8rj8\") pod \"maas-api-key-cleanup-29895540-b4sms\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.387339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387229 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.387339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387268 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkcq4\" (UniqueName: \"kubernetes.io/projected/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-kube-api-access-pkcq4\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.387339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387298 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.387850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387819 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.387991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.387854 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7588e6c6-b872-4193-8de8-990012b2e689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.388117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.388098 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.388117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.388106 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.388232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.388222 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dff7e1-48b6-44b9-a59e-5f409d535387-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-b4sms\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.459374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.459339 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhkb\" (UniqueName: \"kubernetes.io/projected/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-kube-api-access-wxhkb\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.460062 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.460033 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvkd\" (UniqueName: \"kubernetes.io/projected/7588e6c6-b872-4193-8de8-990012b2e689-kube-api-access-csvkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.460194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.460074 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkcq4\" (UniqueName: \"kubernetes.io/projected/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-kube-api-access-pkcq4\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.460314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.460291 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rj8\" (UniqueName: \"kubernetes.io/projected/01dff7e1-48b6-44b9-a59e-5f409d535387-kube-api-access-p8rj8\") pod \"maas-api-key-cleanup-29895540-b4sms\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.460603 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.460584 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59xz\" (UniqueName: \"kubernetes.io/projected/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-kube-api-access-f59xz\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.503202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.503174 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:00.517541 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.517486 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:00.524653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.524628 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:00.546860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.546744 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:00.557055 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.557030 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:00.897586 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:00.897556 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dff7e1_48b6_44b9_a59e_5f409d535387.slice/crio-41c3b2dc4c7fd348c8a72c1a6ca4a63d87bf5a8f96fc70d6a0c178feb759b482 WatchSource:0}: Error finding container 41c3b2dc4c7fd348c8a72c1a6ca4a63d87bf5a8f96fc70d6a0c178feb759b482: Status 404 returned error can't find the container with id 41c3b2dc4c7fd348c8a72c1a6ca4a63d87bf5a8f96fc70d6a0c178feb759b482 Nov 03 19:00:00.900532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.900511 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms"] Nov 03 19:00:00.945602 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:00.945579 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7588e6c6_b872_4193_8de8_990012b2e689.slice/crio-8f6bc3e0680ce6a62f6bf231178f2e90dc97a088125986227353ec374c044fd6 WatchSource:0}: Error finding container 8f6bc3e0680ce6a62f6bf231178f2e90dc97a088125986227353ec374c044fd6: Status 404 returned error can't find the container with id 8f6bc3e0680ce6a62f6bf231178f2e90dc97a088125986227353ec374c044fd6 Nov 03 19:00:00.946572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.946549 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl"] Nov 03 19:00:00.961311 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:00.961255 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b0898a_75ac_43ff_a85c_61c0e55da2ea.slice/crio-ca78f8b03a58b3787a92f6ca03b086e1b5bfd458d9843d747b23e69afc9f198b WatchSource:0}: Error finding container ca78f8b03a58b3787a92f6ca03b086e1b5bfd458d9843d747b23e69afc9f198b: Status 404 returned error can't find the container with id ca78f8b03a58b3787a92f6ca03b086e1b5bfd458d9843d747b23e69afc9f198b Nov 03 19:00:00.962352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:00.962262 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g"] Nov 03 19:00:01.028744 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:01.028715 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f48414_5c98_4ee8_9e7f_45cd3f1db6a8.slice/crio-07f6a5c01deb12586d4bd18f68d822b7f32314974baf43115db65ca86fb76141 WatchSource:0}: Error finding container 07f6a5c01deb12586d4bd18f68d822b7f32314974baf43115db65ca86fb76141: Status 404 returned error can't find the container with id 07f6a5c01deb12586d4bd18f68d822b7f32314974baf43115db65ca86fb76141 Nov 03 19:00:01.029257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:01.029234 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc"] Nov 03 19:00:01.033014 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:01.032978 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a86af3_b752_4ba3_99b2_ceb6aedd9c74.slice/crio-4dc4972c2185436165c232572ca52248a3870569fb951170373a704a695226f5 WatchSource:0}: Error finding container 4dc4972c2185436165c232572ca52248a3870569fb951170373a704a695226f5: Status 404 returned error can't find the container with id 4dc4972c2185436165c232572ca52248a3870569fb951170373a704a695226f5 Nov 03 19:00:01.035370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:01.035351 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b"] Nov 03 19:00:01.128949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:01.128910 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" event={"ID":"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8","Type":"ContainerStarted","Data":"07f6a5c01deb12586d4bd18f68d822b7f32314974baf43115db65ca86fb76141"} Nov 03 19:00:01.130543 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:01.130511 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" event={"ID":"01dff7e1-48b6-44b9-a59e-5f409d535387","Type":"ContainerStarted","Data":"41c3b2dc4c7fd348c8a72c1a6ca4a63d87bf5a8f96fc70d6a0c178feb759b482"} Nov 03 19:00:01.131928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:01.131904 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" event={"ID":"f0b0898a-75ac-43ff-a85c-61c0e55da2ea","Type":"ContainerStarted","Data":"ca78f8b03a58b3787a92f6ca03b086e1b5bfd458d9843d747b23e69afc9f198b"} Nov 03 19:00:01.133209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:01.133186 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" event={"ID":"7588e6c6-b872-4193-8de8-990012b2e689","Type":"ContainerStarted","Data":"8f6bc3e0680ce6a62f6bf231178f2e90dc97a088125986227353ec374c044fd6"} Nov 03 19:00:01.134405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:01.134383 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerStarted","Data":"4dc4972c2185436165c232572ca52248a3870569fb951170373a704a695226f5"} Nov 03 19:00:02.144843 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.144803 4948 generic.go:356] "Generic (PLEG): container finished" podID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerID="b90d876c95e03dc57e162496f3d23696400c662c886d11a8b056f3aa670de43b" exitCode=60 Nov 03 19:00:02.145297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.144898 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" event={"ID":"01dff7e1-48b6-44b9-a59e-5f409d535387","Type":"ContainerDied","Data":"b90d876c95e03dc57e162496f3d23696400c662c886d11a8b056f3aa670de43b"} Nov 03 19:00:02.145297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.145168 4948 scope.go:122] "RemoveContainer" containerID="b90d876c95e03dc57e162496f3d23696400c662c886d11a8b056f3aa670de43b" Nov 03 19:00:02.146772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.146749 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerID="a7d0c2f1d5970929634c3ac54de8310eaf2e9bf5adcf3a25f730844820dc20c7" exitCode=60 Nov 03 19:00:02.146928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.146831 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" event={"ID":"f0b0898a-75ac-43ff-a85c-61c0e55da2ea","Type":"ContainerDied","Data":"a7d0c2f1d5970929634c3ac54de8310eaf2e9bf5adcf3a25f730844820dc20c7"} Nov 03 19:00:02.147067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.147050 4948 scope.go:122] "RemoveContainer" containerID="a7d0c2f1d5970929634c3ac54de8310eaf2e9bf5adcf3a25f730844820dc20c7" Nov 03 19:00:02.149025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.148880 4948 generic.go:356] "Generic (PLEG): container finished" podID="7588e6c6-b872-4193-8de8-990012b2e689" containerID="15bdad818b14a6bb3e6567b84e1110b67a39b2155ab14f46cae033b3e45fa77e" exitCode=60 Nov 03 19:00:02.149025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.148959 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" event={"ID":"7588e6c6-b872-4193-8de8-990012b2e689","Type":"ContainerDied","Data":"15bdad818b14a6bb3e6567b84e1110b67a39b2155ab14f46cae033b3e45fa77e"} Nov 03 19:00:02.149172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.149156 4948 scope.go:122] "RemoveContainer" containerID="15bdad818b14a6bb3e6567b84e1110b67a39b2155ab14f46cae033b3e45fa77e" Nov 03 19:00:02.150512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.150493 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerStarted","Data":"b97bb29e03290fa334bdc98f3f08f5c2a95b8401594f3baa8ef5f758fdc84f8d"} Nov 03 19:00:02.152488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.152467 4948 generic.go:356] "Generic (PLEG): container finished" podID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerID="0bb38835bec8628cbb50a3dd1030cdc985c9cbc9fc1d7277adc72bb6de626ad1" exitCode=60 Nov 03 19:00:02.152607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.152518 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" event={"ID":"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8","Type":"ContainerDied","Data":"0bb38835bec8628cbb50a3dd1030cdc985c9cbc9fc1d7277adc72bb6de626ad1"} Nov 03 19:00:02.152798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.152728 4948 scope.go:122] "RemoveContainer" containerID="0bb38835bec8628cbb50a3dd1030cdc985c9cbc9fc1d7277adc72bb6de626ad1" Nov 03 19:00:02.786037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:02.785982 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" podStartSLOduration=2.7859679870000003 podStartE2EDuration="2.785967987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:02.785895 +0000 UTC m=+1332.268890913" watchObservedRunningTime="2026-11-03 19:00:02.785967987 +0000 UTC m=+1332.268963900" Nov 03 19:00:02.805866 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:02.805830 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:00:03.165592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.165559 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerID="b97bb29e03290fa334bdc98f3f08f5c2a95b8401594f3baa8ef5f758fdc84f8d" exitCode=60 Nov 03 19:00:03.166085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.165639 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerDied","Data":"b97bb29e03290fa334bdc98f3f08f5c2a95b8401594f3baa8ef5f758fdc84f8d"} Nov 03 19:00:03.166085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.165949 4948 scope.go:122] "RemoveContainer" containerID="b97bb29e03290fa334bdc98f3f08f5c2a95b8401594f3baa8ef5f758fdc84f8d" Nov 03 19:00:03.168309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.168282 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" event={"ID":"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8","Type":"ContainerStarted","Data":"df3788bec80f4912805fcc2486ca7528a08f56a1e080f43ea03cf7a27072a8c3"} Nov 03 19:00:03.173511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.173487 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" event={"ID":"01dff7e1-48b6-44b9-a59e-5f409d535387","Type":"ContainerStarted","Data":"f36edb4f7a0e54cb0ecaa3d52a41ab281a53bebd62e15ea869a9409a241a3ab5"} Nov 03 19:00:03.175824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.175795 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" event={"ID":"f0b0898a-75ac-43ff-a85c-61c0e55da2ea","Type":"ContainerStarted","Data":"7b172502149caea008d05ea4d137b439a36e5e2481b5fd17ced4486f878cc0f0"} Nov 03 19:00:03.177807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.177786 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" event={"ID":"7588e6c6-b872-4193-8de8-990012b2e689","Type":"ContainerStarted","Data":"20d728308a058eda01b20c15da4da1f568020da42ccb4f4ca861bfe9302b290c"} Nov 03 19:00:03.406099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.406045 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" podStartSLOduration=3.4060309269999998 podStartE2EDuration="3.406030927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:03.401384247 +0000 UTC m=+1332.884380162" watchObservedRunningTime="2026-11-03 19:00:03.406030927 +0000 UTC m=+1332.889026840" Nov 03 19:00:03.523747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.523685 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" podStartSLOduration=3.523670513 podStartE2EDuration="3.523670513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:03.519534149 +0000 UTC m=+1333.002530063" watchObservedRunningTime="2026-11-03 19:00:03.523670513 +0000 UTC m=+1333.006666443" Nov 03 19:00:03.644191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.644124 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" podStartSLOduration=3.644103779 podStartE2EDuration="3.644103779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:03.636689273 +0000 UTC m=+1333.119685188" watchObservedRunningTime="2026-11-03 19:00:03.644103779 +0000 UTC m=+1333.127099694" Nov 03 19:00:03.756444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:03.756336 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" podStartSLOduration=3.756319627 podStartE2EDuration="3.756319627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:03.75417397 +0000 UTC m=+1333.237169887" watchObservedRunningTime="2026-11-03 19:00:03.756319627 +0000 UTC m=+1333.239315540" Nov 03 19:00:04.191722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.191655 4948 generic.go:356] "Generic (PLEG): container finished" podID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerID="f36edb4f7a0e54cb0ecaa3d52a41ab281a53bebd62e15ea869a9409a241a3ab5" exitCode=60 Nov 03 19:00:04.192186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.191781 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" event={"ID":"01dff7e1-48b6-44b9-a59e-5f409d535387","Type":"ContainerDied","Data":"f36edb4f7a0e54cb0ecaa3d52a41ab281a53bebd62e15ea869a9409a241a3ab5"} Nov 03 19:00:04.192186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.191863 4948 scope.go:122] "RemoveContainer" containerID="b90d876c95e03dc57e162496f3d23696400c662c886d11a8b056f3aa670de43b" Nov 03 19:00:04.192314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.192219 4948 scope.go:122] "RemoveContainer" containerID="f36edb4f7a0e54cb0ecaa3d52a41ab281a53bebd62e15ea869a9409a241a3ab5" Nov 03 19:00:04.192626 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:04.192602 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-b4sms_odh-ai-gateway-infra(01dff7e1-48b6-44b9-a59e-5f409d535387)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" Nov 03 19:00:04.201438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.201407 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerID="7b172502149caea008d05ea4d137b439a36e5e2481b5fd17ced4486f878cc0f0" exitCode=60 Nov 03 19:00:04.201569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.201491 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" event={"ID":"f0b0898a-75ac-43ff-a85c-61c0e55da2ea","Type":"ContainerDied","Data":"7b172502149caea008d05ea4d137b439a36e5e2481b5fd17ced4486f878cc0f0"} Nov 03 19:00:04.201824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.201801 4948 scope.go:122] "RemoveContainer" containerID="7b172502149caea008d05ea4d137b439a36e5e2481b5fd17ced4486f878cc0f0" Nov 03 19:00:04.202156 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:04.202122 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g_odh-ai-gateway-infra(f0b0898a-75ac-43ff-a85c-61c0e55da2ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" Nov 03 19:00:04.203856 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.203830 4948 generic.go:356] "Generic (PLEG): container finished" podID="7588e6c6-b872-4193-8de8-990012b2e689" containerID="20d728308a058eda01b20c15da4da1f568020da42ccb4f4ca861bfe9302b290c" exitCode=60 Nov 03 19:00:04.203947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.203883 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" event={"ID":"7588e6c6-b872-4193-8de8-990012b2e689","Type":"ContainerDied","Data":"20d728308a058eda01b20c15da4da1f568020da42ccb4f4ca861bfe9302b290c"} Nov 03 19:00:04.204194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.204177 4948 scope.go:122] "RemoveContainer" containerID="20d728308a058eda01b20c15da4da1f568020da42ccb4f4ca861bfe9302b290c" Nov 03 19:00:04.204518 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:04.204493 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl_odh-ai-gateway-infra(7588e6c6-b872-4193-8de8-990012b2e689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" podUID="7588e6c6-b872-4193-8de8-990012b2e689" Nov 03 19:00:04.206181 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.206157 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerStarted","Data":"9a328a8a3b89c30d9bf4bb06eeef297f41c816f60fe0d17af58e8a0bef0c57f5"} Nov 03 19:00:04.208235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.208215 4948 generic.go:356] "Generic (PLEG): container finished" podID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerID="df3788bec80f4912805fcc2486ca7528a08f56a1e080f43ea03cf7a27072a8c3" exitCode=60 Nov 03 19:00:04.208332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.208260 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" event={"ID":"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8","Type":"ContainerDied","Data":"df3788bec80f4912805fcc2486ca7528a08f56a1e080f43ea03cf7a27072a8c3"} Nov 03 19:00:04.208447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.208435 4948 scope.go:122] "RemoveContainer" containerID="df3788bec80f4912805fcc2486ca7528a08f56a1e080f43ea03cf7a27072a8c3" Nov 03 19:00:04.208633 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:04.208618 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc_odh-ai-gateway-infra(e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" Nov 03 19:00:04.212196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.212177 4948 scope.go:122] "RemoveContainer" containerID="a7d0c2f1d5970929634c3ac54de8310eaf2e9bf5adcf3a25f730844820dc20c7" Nov 03 19:00:04.226360 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.226342 4948 scope.go:122] "RemoveContainer" containerID="15bdad818b14a6bb3e6567b84e1110b67a39b2155ab14f46cae033b3e45fa77e" Nov 03 19:00:04.240583 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:04.240563 4948 scope.go:122] "RemoveContainer" containerID="0bb38835bec8628cbb50a3dd1030cdc985c9cbc9fc1d7277adc72bb6de626ad1" Nov 03 19:00:05.221104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.221075 4948 scope.go:122] "RemoveContainer" containerID="df3788bec80f4912805fcc2486ca7528a08f56a1e080f43ea03cf7a27072a8c3" Nov 03 19:00:05.221637 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:05.221393 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc_odh-ai-gateway-infra(e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" Nov 03 19:00:05.222948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.222923 4948 scope.go:122] "RemoveContainer" containerID="f36edb4f7a0e54cb0ecaa3d52a41ab281a53bebd62e15ea869a9409a241a3ab5" Nov 03 19:00:05.223202 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:05.223182 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-b4sms_odh-ai-gateway-infra(01dff7e1-48b6-44b9-a59e-5f409d535387)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" Nov 03 19:00:05.225078 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.225061 4948 scope.go:122] "RemoveContainer" containerID="7b172502149caea008d05ea4d137b439a36e5e2481b5fd17ced4486f878cc0f0" Nov 03 19:00:05.225306 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:05.225279 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g_odh-ai-gateway-infra(f0b0898a-75ac-43ff-a85c-61c0e55da2ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" Nov 03 19:00:05.226842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.226822 4948 scope.go:122] "RemoveContainer" containerID="20d728308a058eda01b20c15da4da1f568020da42ccb4f4ca861bfe9302b290c" Nov 03 19:00:05.227062 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:05.227042 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl_odh-ai-gateway-infra(7588e6c6-b872-4193-8de8-990012b2e689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" podUID="7588e6c6-b872-4193-8de8-990012b2e689" Nov 03 19:00:05.228605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.228581 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerID="9a328a8a3b89c30d9bf4bb06eeef297f41c816f60fe0d17af58e8a0bef0c57f5" exitCode=60 Nov 03 19:00:05.228686 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.228658 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerDied","Data":"9a328a8a3b89c30d9bf4bb06eeef297f41c816f60fe0d17af58e8a0bef0c57f5"} Nov 03 19:00:05.228753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.228740 4948 scope.go:122] "RemoveContainer" containerID="b97bb29e03290fa334bdc98f3f08f5c2a95b8401594f3baa8ef5f758fdc84f8d" Nov 03 19:00:05.228878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:05.228858 4948 scope.go:122] "RemoveContainer" containerID="9a328a8a3b89c30d9bf4bb06eeef297f41c816f60fe0d17af58e8a0bef0c57f5" Nov 03 19:00:05.229118 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:05.229089 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b_odh-ai-gateway-infra(f0a86af3-b752-4ba3-99b2-ceb6aedd9c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" Nov 03 19:00:06.243467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:06.243438 4948 scope.go:122] "RemoveContainer" containerID="9a328a8a3b89c30d9bf4bb06eeef297f41c816f60fe0d17af58e8a0bef0c57f5" Nov 03 19:00:06.243968 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:06.243839 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b_odh-ai-gateway-infra(f0a86af3-b752-4ba3-99b2-ceb6aedd9c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" Nov 03 19:00:08.390146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.390113 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t"] Nov 03 19:00:08.394524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.394508 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.449054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.449026 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t"] Nov 03 19:00:08.494369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.494341 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:00:08.514245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.514209 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.514382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.514281 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvb5d\" (UniqueName: \"kubernetes.io/projected/08fcf34c-d285-4be6-ac9b-be88bc4f457f-kube-api-access-wvb5d\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.514382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.514371 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.615068 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.615040 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.615220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.615167 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.615220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.615190 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvb5d\" (UniqueName: \"kubernetes.io/projected/08fcf34c-d285-4be6-ac9b-be88bc4f457f-kube-api-access-wvb5d\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.615851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.615807 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.617581 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.617556 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.681174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.681102 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvb5d\" (UniqueName: \"kubernetes.io/projected/08fcf34c-d285-4be6-ac9b-be88bc4f457f-kube-api-access-wvb5d\") pod \"maas-api-revoke-sub-b4e23c0d-m492t\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:08.708530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:08.708504 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:09.093286 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:09.093258 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fcf34c_d285_4be6_ac9b_be88bc4f457f.slice/crio-682072c3dbbbb3a383d0031349ea37d50ddd4b9786b13becf14a166e97353cae WatchSource:0}: Error finding container 682072c3dbbbb3a383d0031349ea37d50ddd4b9786b13becf14a166e97353cae: Status 404 returned error can't find the container with id 682072c3dbbbb3a383d0031349ea37d50ddd4b9786b13becf14a166e97353cae Nov 03 19:00:09.093853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:09.093832 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t"] Nov 03 19:00:09.325074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:09.325043 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" event={"ID":"08fcf34c-d285-4be6-ac9b-be88bc4f457f","Type":"ContainerStarted","Data":"682072c3dbbbb3a383d0031349ea37d50ddd4b9786b13becf14a166e97353cae"} Nov 03 19:00:10.336572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:10.336479 4948 generic.go:356] "Generic (PLEG): container finished" podID="08fcf34c-d285-4be6-ac9b-be88bc4f457f" containerID="7ab1982cc1e6aa7dfea5d92e81ce5f68a9820a3b27cb3e1f823582e305a55cf1" exitCode=0 Nov 03 19:00:10.336985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:10.336564 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" event={"ID":"08fcf34c-d285-4be6-ac9b-be88bc4f457f","Type":"ContainerDied","Data":"7ab1982cc1e6aa7dfea5d92e81ce5f68a9820a3b27cb3e1f823582e305a55cf1"} Nov 03 19:00:11.505064 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.505031 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:11.560505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.560472 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-openshift-ca\") pod \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " Nov 03 19:00:11.560674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.560644 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fcf34c-d285-4be6-ac9b-be88bc4f457f-kube-api-access-wvb5d\" (UniqueName: \"kubernetes.io/projected/08fcf34c-d285-4be6-ac9b-be88bc4f457f-kube-api-access-wvb5d\") pod \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " Nov 03 19:00:11.560760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.560736 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-xks-ca\") pod \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\" (UID: \"08fcf34c-d285-4be6-ac9b-be88bc4f457f\") " Nov 03 19:00:11.560821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.560780 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-openshift-ca" pod "08fcf34c-d285-4be6-ac9b-be88bc4f457f" (UID: "08fcf34c-d285-4be6-ac9b-be88bc4f457f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:11.561186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.561162 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:11.562909 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.562888 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fcf34c-d285-4be6-ac9b-be88bc4f457f-kube-api-access-wvb5d" pod "08fcf34c-d285-4be6-ac9b-be88bc4f457f" (UID: "08fcf34c-d285-4be6-ac9b-be88bc4f457f"). InnerVolumeSpecName "kube-api-access-wvb5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:11.562980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.562914 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-xks-ca" pod "08fcf34c-d285-4be6-ac9b-be88bc4f457f" (UID: "08fcf34c-d285-4be6-ac9b-be88bc4f457f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:00:11.663090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.663012 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvb5d\" (UniqueName: \"kubernetes.io/projected/08fcf34c-d285-4be6-ac9b-be88bc4f457f-kube-api-access-wvb5d\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:11.663090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:11.663043 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fcf34c-d285-4be6-ac9b-be88bc4f457f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:12.354661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:12.354628 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" Nov 03 19:00:12.354661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:12.354650 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t" event={"ID":"08fcf34c-d285-4be6-ac9b-be88bc4f457f","Type":"ContainerDied","Data":"682072c3dbbbb3a383d0031349ea37d50ddd4b9786b13becf14a166e97353cae"} Nov 03 19:00:12.354922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:12.354730 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682072c3dbbbb3a383d0031349ea37d50ddd4b9786b13becf14a166e97353cae" Nov 03 19:00:13.746013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:13.745983 4948 scope.go:122] "RemoveContainer" containerID="df3788bec80f4912805fcc2486ca7528a08f56a1e080f43ea03cf7a27072a8c3" Nov 03 19:00:14.377382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:14.377281 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" event={"ID":"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8","Type":"ContainerStarted","Data":"d3c667ec96dfd60f8d325a07267db5c3eb7557e2541afbe76ed9ea8162ae9338"} Nov 03 19:00:15.390630 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.390596 4948 generic.go:356] "Generic (PLEG): container finished" podID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerID="d3c667ec96dfd60f8d325a07267db5c3eb7557e2541afbe76ed9ea8162ae9338" exitCode=60 Nov 03 19:00:15.391131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.390669 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" event={"ID":"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8","Type":"ContainerDied","Data":"d3c667ec96dfd60f8d325a07267db5c3eb7557e2541afbe76ed9ea8162ae9338"} Nov 03 19:00:15.391131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.390771 4948 scope.go:122] "RemoveContainer" containerID="df3788bec80f4912805fcc2486ca7528a08f56a1e080f43ea03cf7a27072a8c3" Nov 03 19:00:15.391257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.391159 4948 scope.go:122] "RemoveContainer" containerID="d3c667ec96dfd60f8d325a07267db5c3eb7557e2541afbe76ed9ea8162ae9338" Nov 03 19:00:15.391545 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:15.391516 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc_odh-ai-gateway-infra(e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" Nov 03 19:00:15.510989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.510956 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc"] Nov 03 19:00:15.745806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.745772 4948 scope.go:122] "RemoveContainer" containerID="7b172502149caea008d05ea4d137b439a36e5e2481b5fd17ced4486f878cc0f0" Nov 03 19:00:15.923002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.922970 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf"] Nov 03 19:00:15.923577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.923565 4948 cpu_manager.go:414] "containerMap: removing container" podUID="08fcf34c-d285-4be6-ac9b-be88bc4f457f" containerName="revoke-keys" Nov 03 19:00:15.923623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.923578 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fcf34c-d285-4be6-ac9b-be88bc4f457f" containerName="revoke-keys" Nov 03 19:00:15.923731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.923720 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fcf34c-d285-4be6-ac9b-be88bc4f457f" containerName="revoke-keys" Nov 03 19:00:15.928723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.928675 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:15.984536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.984481 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"distinct-2-w6-fba266-kserve-self-signed-certs\"" Nov 03 19:00:15.984922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:15.984901 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf"] Nov 03 19:00:16.023017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.022948 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-tmp-dir\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.023190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.023169 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afb714-a01e-4660-946a-71ae6568adaa-tls-certs\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.023234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.023206 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-home\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.023283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.023237 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-model-cache\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.023323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.023293 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzvz\" (UniqueName: \"kubernetes.io/projected/c9afb714-a01e-4660-946a-71ae6568adaa-kube-api-access-swzvz\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.023387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.023365 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-dshm\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.124715 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afb714-a01e-4660-946a-71ae6568adaa-tls-certs\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.124770 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-home\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.124795 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-model-cache\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.124829 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzvz\" (UniqueName: \"kubernetes.io/projected/c9afb714-a01e-4660-946a-71ae6568adaa-kube-api-access-swzvz\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.124870 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-dshm\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.124968 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-tmp-dir\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.125541 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-tmp-dir\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.126235 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-model-cache\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.126742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.126479 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-home\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.130475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.130438 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afb714-a01e-4660-946a-71ae6568adaa-tls-certs\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.131390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.131358 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-dshm\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.197401 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.197370 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzvz\" (UniqueName: \"kubernetes.io/projected/c9afb714-a01e-4660-946a-71ae6568adaa-kube-api-access-swzvz\") pod \"distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.241255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.241220 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:16.418627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.418353 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" event={"ID":"f0b0898a-75ac-43ff-a85c-61c0e55da2ea","Type":"ContainerStarted","Data":"47f3ed6a07675563e70265c9c68c753f5f21511d84f8516288eb2525ff8653c6"} Nov 03 19:00:16.589100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.589074 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:16.707690 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:16.707663 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9afb714_a01e_4660_946a_71ae6568adaa.slice/crio-165b46f8b47dbc84bbcb845d692e30f394bc8dcf2693ef615e5933efae3dcc5a WatchSource:0}: Error finding container 165b46f8b47dbc84bbcb845d692e30f394bc8dcf2693ef615e5933efae3dcc5a: Status 404 returned error can't find the container with id 165b46f8b47dbc84bbcb845d692e30f394bc8dcf2693ef615e5933efae3dcc5a Nov 03 19:00:16.709572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.709533 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf"] Nov 03 19:00:16.732996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.732973 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-maas-api-service-ca\") pod \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " Nov 03 19:00:16.733103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.733089 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-kube-api-access-f59xz\" (UniqueName: \"kubernetes.io/projected/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-kube-api-access-f59xz\") pod \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\" (UID: \"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8\") " Nov 03 19:00:16.733387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.733359 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-maas-api-service-ca" pod "e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" (UID: "e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:16.733532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.733516 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:16.735002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.734983 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-kube-api-access-f59xz" pod "e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" (UID: "e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8"). InnerVolumeSpecName "kube-api-access-f59xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:16.746477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.746447 4948 scope.go:122] "RemoveContainer" containerID="20d728308a058eda01b20c15da4da1f568020da42ccb4f4ca861bfe9302b290c" Nov 03 19:00:16.746572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.746518 4948 scope.go:122] "RemoveContainer" containerID="f36edb4f7a0e54cb0ecaa3d52a41ab281a53bebd62e15ea869a9409a241a3ab5" Nov 03 19:00:16.835471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:16.835168 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59xz\" (UniqueName: \"kubernetes.io/projected/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8-kube-api-access-f59xz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:17.431954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.431857 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" event={"ID":"7588e6c6-b872-4193-8de8-990012b2e689","Type":"ContainerStarted","Data":"8021947386e2b2c879e18ebddf716fa9a3dbddc8984dd5b6bb7b5a4edf09f3e9"} Nov 03 19:00:17.433672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.433641 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" event={"ID":"c9afb714-a01e-4660-946a-71ae6568adaa","Type":"ContainerStarted","Data":"58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb"} Nov 03 19:00:17.433856 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.433684 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" event={"ID":"c9afb714-a01e-4660-946a-71ae6568adaa","Type":"ContainerStarted","Data":"165b46f8b47dbc84bbcb845d692e30f394bc8dcf2693ef615e5933efae3dcc5a"} Nov 03 19:00:17.435363 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.435345 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" Nov 03 19:00:17.435485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.435384 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc" event={"ID":"e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8","Type":"ContainerDied","Data":"07f6a5c01deb12586d4bd18f68d822b7f32314974baf43115db65ca86fb76141"} Nov 03 19:00:17.435485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.435443 4948 scope.go:122] "RemoveContainer" containerID="d3c667ec96dfd60f8d325a07267db5c3eb7557e2541afbe76ed9ea8162ae9338" Nov 03 19:00:17.438028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.438006 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" event={"ID":"01dff7e1-48b6-44b9-a59e-5f409d535387","Type":"ContainerStarted","Data":"c5d7a5aa982767bd7ce5fe23a57b6b931d777518c25b892d79818deff1f2ef37"} Nov 03 19:00:17.440272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.440251 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerID="47f3ed6a07675563e70265c9c68c753f5f21511d84f8516288eb2525ff8653c6" exitCode=60 Nov 03 19:00:17.440341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.440284 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" event={"ID":"f0b0898a-75ac-43ff-a85c-61c0e55da2ea","Type":"ContainerDied","Data":"47f3ed6a07675563e70265c9c68c753f5f21511d84f8516288eb2525ff8653c6"} Nov 03 19:00:17.440537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.440518 4948 scope.go:122] "RemoveContainer" containerID="47f3ed6a07675563e70265c9c68c753f5f21511d84f8516288eb2525ff8653c6" Nov 03 19:00:17.440786 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:17.440766 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g_odh-ai-gateway-infra(f0b0898a-75ac-43ff-a85c-61c0e55da2ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" Nov 03 19:00:17.447344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.447322 4948 scope.go:122] "RemoveContainer" containerID="7b172502149caea008d05ea4d137b439a36e5e2481b5fd17ced4486f878cc0f0" Nov 03 19:00:17.555561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.555469 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g"] Nov 03 19:00:17.679579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.679542 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc"] Nov 03 19:00:17.687189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.687160 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895540-ts2gc"] Nov 03 19:00:17.802231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:17.802163 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podStartSLOduration=2.802143707 podStartE2EDuration="2.802143707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:17.800522612 +0000 UTC m=+1347.283518527" watchObservedRunningTime="2026-11-03 19:00:17.802143707 +0000 UTC m=+1347.285139621" Nov 03 19:00:18.242053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.242023 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:18.453785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.453746 4948 generic.go:356] "Generic (PLEG): container finished" podID="7588e6c6-b872-4193-8de8-990012b2e689" containerID="8021947386e2b2c879e18ebddf716fa9a3dbddc8984dd5b6bb7b5a4edf09f3e9" exitCode=60 Nov 03 19:00:18.454290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.453895 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" event={"ID":"7588e6c6-b872-4193-8de8-990012b2e689","Type":"ContainerDied","Data":"8021947386e2b2c879e18ebddf716fa9a3dbddc8984dd5b6bb7b5a4edf09f3e9"} Nov 03 19:00:18.454290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.453966 4948 scope.go:122] "RemoveContainer" containerID="20d728308a058eda01b20c15da4da1f568020da42ccb4f4ca861bfe9302b290c" Nov 03 19:00:18.454380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.454364 4948 scope.go:122] "RemoveContainer" containerID="8021947386e2b2c879e18ebddf716fa9a3dbddc8984dd5b6bb7b5a4edf09f3e9" Nov 03 19:00:18.454799 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:18.454763 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl_odh-ai-gateway-infra(7588e6c6-b872-4193-8de8-990012b2e689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" podUID="7588e6c6-b872-4193-8de8-990012b2e689" Nov 03 19:00:18.458024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.458001 4948 generic.go:356] "Generic (PLEG): container finished" podID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerID="c5d7a5aa982767bd7ce5fe23a57b6b931d777518c25b892d79818deff1f2ef37" exitCode=60 Nov 03 19:00:18.458135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.458037 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" event={"ID":"01dff7e1-48b6-44b9-a59e-5f409d535387","Type":"ContainerDied","Data":"c5d7a5aa982767bd7ce5fe23a57b6b931d777518c25b892d79818deff1f2ef37"} Nov 03 19:00:18.458370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.458356 4948 scope.go:122] "RemoveContainer" containerID="c5d7a5aa982767bd7ce5fe23a57b6b931d777518c25b892d79818deff1f2ef37" Nov 03 19:00:18.458661 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:18.458627 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-b4sms_odh-ai-gateway-infra(01dff7e1-48b6-44b9-a59e-5f409d535387)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" Nov 03 19:00:18.472139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.472119 4948 scope.go:122] "RemoveContainer" containerID="f36edb4f7a0e54cb0ecaa3d52a41ab281a53bebd62e15ea869a9409a241a3ab5" Nov 03 19:00:18.571717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.570935 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl"] Nov 03 19:00:18.633611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.633586 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:18.746572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.746544 4948 scope.go:122] "RemoveContainer" containerID="9a328a8a3b89c30d9bf4bb06eeef297f41c816f60fe0d17af58e8a0bef0c57f5" Nov 03 19:00:18.757666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.757635 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" path="/var/lib/kubelet/pods/e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8/volumes" Nov 03 19:00:18.758712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.758675 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-kube-api-access-wxhkb\" (UniqueName: \"kubernetes.io/projected/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-kube-api-access-wxhkb\") pod \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " Nov 03 19:00:18.758883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.758861 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-maas-api-service-ca\") pod \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\" (UID: \"f0b0898a-75ac-43ff-a85c-61c0e55da2ea\") " Nov 03 19:00:18.759562 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.759535 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-maas-api-service-ca" pod "f0b0898a-75ac-43ff-a85c-61c0e55da2ea" (UID: "f0b0898a-75ac-43ff-a85c-61c0e55da2ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:18.762254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.762233 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-kube-api-access-wxhkb" pod "f0b0898a-75ac-43ff-a85c-61c0e55da2ea" (UID: "f0b0898a-75ac-43ff-a85c-61c0e55da2ea"). InnerVolumeSpecName "kube-api-access-wxhkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:18.860548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.860517 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxhkb\" (UniqueName: \"kubernetes.io/projected/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-kube-api-access-wxhkb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:18.860548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.860545 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b0898a-75ac-43ff-a85c-61c0e55da2ea-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:18.934721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:18.934668 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms"] Nov 03 19:00:19.473058 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.473028 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" Nov 03 19:00:19.473469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.473043 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g" event={"ID":"f0b0898a-75ac-43ff-a85c-61c0e55da2ea","Type":"ContainerDied","Data":"ca78f8b03a58b3787a92f6ca03b086e1b5bfd458d9843d747b23e69afc9f198b"} Nov 03 19:00:19.473469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.473153 4948 scope.go:122] "RemoveContainer" containerID="47f3ed6a07675563e70265c9c68c753f5f21511d84f8516288eb2525ff8653c6" Nov 03 19:00:19.480394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.479602 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerStarted","Data":"395e42c19e8b22b6ad98f6f98107eb8d167023b1e4f82488e8a79d0cd1a4ff2e"} Nov 03 19:00:19.676434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.676397 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:19.707904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.707879 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g"] Nov 03 19:00:19.710837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.710816 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:19.712838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.712817 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895540-b4x8g"] Nov 03 19:00:19.773316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.773285 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7588e6c6-b872-4193-8de8-990012b2e689-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7588e6c6-b872-4193-8de8-990012b2e689-maas-api-service-ca\") pod \"7588e6c6-b872-4193-8de8-990012b2e689\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " Nov 03 19:00:19.773451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.773328 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7588e6c6-b872-4193-8de8-990012b2e689-kube-api-access-csvkd\" (UniqueName: \"kubernetes.io/projected/7588e6c6-b872-4193-8de8-990012b2e689-kube-api-access-csvkd\") pod \"7588e6c6-b872-4193-8de8-990012b2e689\" (UID: \"7588e6c6-b872-4193-8de8-990012b2e689\") " Nov 03 19:00:19.773721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.773678 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7588e6c6-b872-4193-8de8-990012b2e689-maas-api-service-ca" pod "7588e6c6-b872-4193-8de8-990012b2e689" (UID: "7588e6c6-b872-4193-8de8-990012b2e689"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:19.773919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.773894 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7588e6c6-b872-4193-8de8-990012b2e689-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:19.775469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.775444 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7588e6c6-b872-4193-8de8-990012b2e689-kube-api-access-csvkd" pod "7588e6c6-b872-4193-8de8-990012b2e689" (UID: "7588e6c6-b872-4193-8de8-990012b2e689"). InnerVolumeSpecName "kube-api-access-csvkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:19.874940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.874910 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dff7e1-48b6-44b9-a59e-5f409d535387-kube-api-access-p8rj8\" (UniqueName: \"kubernetes.io/projected/01dff7e1-48b6-44b9-a59e-5f409d535387-kube-api-access-p8rj8\") pod \"01dff7e1-48b6-44b9-a59e-5f409d535387\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " Nov 03 19:00:19.875112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.875033 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dff7e1-48b6-44b9-a59e-5f409d535387-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dff7e1-48b6-44b9-a59e-5f409d535387-maas-api-service-ca\") pod \"01dff7e1-48b6-44b9-a59e-5f409d535387\" (UID: \"01dff7e1-48b6-44b9-a59e-5f409d535387\") " Nov 03 19:00:19.876072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.875437 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dff7e1-48b6-44b9-a59e-5f409d535387-maas-api-service-ca" pod "01dff7e1-48b6-44b9-a59e-5f409d535387" (UID: "01dff7e1-48b6-44b9-a59e-5f409d535387"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:19.876072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.875846 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dff7e1-48b6-44b9-a59e-5f409d535387-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:19.876072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.875873 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csvkd\" (UniqueName: \"kubernetes.io/projected/7588e6c6-b872-4193-8de8-990012b2e689-kube-api-access-csvkd\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:19.877634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.877611 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dff7e1-48b6-44b9-a59e-5f409d535387-kube-api-access-p8rj8" pod "01dff7e1-48b6-44b9-a59e-5f409d535387" (UID: "01dff7e1-48b6-44b9-a59e-5f409d535387"). InnerVolumeSpecName "kube-api-access-p8rj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:19.976829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:19.976787 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rj8\" (UniqueName: \"kubernetes.io/projected/01dff7e1-48b6-44b9-a59e-5f409d535387-kube-api-access-p8rj8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:20.493239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.493202 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" Nov 03 19:00:20.493735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.493205 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms" event={"ID":"01dff7e1-48b6-44b9-a59e-5f409d535387","Type":"ContainerDied","Data":"41c3b2dc4c7fd348c8a72c1a6ca4a63d87bf5a8f96fc70d6a0c178feb759b482"} Nov 03 19:00:20.493735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.493350 4948 scope.go:122] "RemoveContainer" containerID="c5d7a5aa982767bd7ce5fe23a57b6b931d777518c25b892d79818deff1f2ef37" Nov 03 19:00:20.496264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.496246 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" Nov 03 19:00:20.496380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.496273 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl" event={"ID":"7588e6c6-b872-4193-8de8-990012b2e689","Type":"ContainerDied","Data":"8f6bc3e0680ce6a62f6bf231178f2e90dc97a088125986227353ec374c044fd6"} Nov 03 19:00:20.498342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.498320 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerID="395e42c19e8b22b6ad98f6f98107eb8d167023b1e4f82488e8a79d0cd1a4ff2e" exitCode=60 Nov 03 19:00:20.498510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.498494 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerDied","Data":"395e42c19e8b22b6ad98f6f98107eb8d167023b1e4f82488e8a79d0cd1a4ff2e"} Nov 03 19:00:20.498868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.498852 4948 scope.go:122] "RemoveContainer" containerID="395e42c19e8b22b6ad98f6f98107eb8d167023b1e4f82488e8a79d0cd1a4ff2e" Nov 03 19:00:20.499110 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:20.499091 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b_odh-ai-gateway-infra(f0a86af3-b752-4ba3-99b2-ceb6aedd9c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" Nov 03 19:00:20.507596 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.507579 4948 scope.go:122] "RemoveContainer" containerID="8021947386e2b2c879e18ebddf716fa9a3dbddc8984dd5b6bb7b5a4edf09f3e9" Nov 03 19:00:20.517263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.517244 4948 scope.go:122] "RemoveContainer" containerID="9a328a8a3b89c30d9bf4bb06eeef297f41c816f60fe0d17af58e8a0bef0c57f5" Nov 03 19:00:20.669142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.669107 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms"] Nov 03 19:00:20.676444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.676418 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-b4sms"] Nov 03 19:00:20.756321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.756240 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" path="/var/lib/kubelet/pods/01dff7e1-48b6-44b9-a59e-5f409d535387/volumes" Nov 03 19:00:20.756730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.756715 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" path="/var/lib/kubelet/pods/f0b0898a-75ac-43ff-a85c-61c0e55da2ea/volumes" Nov 03 19:00:20.837612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.837577 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b"] Nov 03 19:00:20.916534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.916498 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl"] Nov 03 19:00:20.920855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:20.920832 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895540-r9pwl"] Nov 03 19:00:21.062528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.062432 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf"] Nov 03 19:00:21.067254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.067220 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deef9c0-l7qdf"] Nov 03 19:00:21.689875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.689845 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:21.802962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.802871 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-maas-api-service-ca\") pod \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " Nov 03 19:00:21.803138 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.803019 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-kube-api-access-pkcq4\" (UniqueName: \"kubernetes.io/projected/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-kube-api-access-pkcq4\") pod \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\" (UID: \"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74\") " Nov 03 19:00:21.803334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.803291 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-maas-api-service-ca" pod "f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" (UID: "f0a86af3-b752-4ba3-99b2-ceb6aedd9c74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:21.803785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.803765 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:21.808509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.808480 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-kube-api-access-pkcq4" pod "f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" (UID: "f0a86af3-b752-4ba3-99b2-ceb6aedd9c74"). InnerVolumeSpecName "kube-api-access-pkcq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:21.922843 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:21.916679 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkcq4\" (UniqueName: \"kubernetes.io/projected/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74-kube-api-access-pkcq4\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:22.524239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.523753 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" event={"ID":"f0a86af3-b752-4ba3-99b2-ceb6aedd9c74","Type":"ContainerDied","Data":"4dc4972c2185436165c232572ca52248a3870569fb951170373a704a695226f5"} Nov 03 19:00:22.524239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.523840 4948 scope.go:122] "RemoveContainer" containerID="395e42c19e8b22b6ad98f6f98107eb8d167023b1e4f82488e8a79d0cd1a4ff2e" Nov 03 19:00:22.524239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.523776 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b" Nov 03 19:00:22.699244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.699211 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b"] Nov 03 19:00:22.706651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.706618 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895540-dzl2b"] Nov 03 19:00:22.754908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.754879 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7588e6c6-b872-4193-8de8-990012b2e689" path="/var/lib/kubelet/pods/7588e6c6-b872-4193-8de8-990012b2e689/volumes" Nov 03 19:00:22.755329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.755316 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" path="/var/lib/kubelet/pods/f0a86af3-b752-4ba3-99b2-ceb6aedd9c74/volumes" Nov 03 19:00:22.755760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:22.755746 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ac0551-c3c5-4275-825e-9618fefd5852" path="/var/lib/kubelet/pods/f0ac0551-c3c5-4275-825e-9618fefd5852/volumes" Nov 03 19:00:26.241607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.241564 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:26.254133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.254106 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:26.301392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.301359 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl"] Nov 03 19:00:26.302321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302301 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.302402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302324 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.302402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302343 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:26.302402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302353 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:26.302402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302365 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:26.302402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302373 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:26.302402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302384 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.302402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302392 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302407 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302414 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302437 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302444 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302461 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302468 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302486 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302492 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302515 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302522 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302532 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302539 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302562 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302571 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302593 4948 cpu_manager.go:414] "containerMap: removing container" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302600 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302614 4948 cpu_manager.go:414] "containerMap: removing container" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.302623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302621 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302632 4948 cpu_manager.go:414] "containerMap: removing container" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302640 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302806 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302820 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302828 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302837 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302846 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302858 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302866 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302875 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302886 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b0898a-75ac-43ff-a85c-61c0e55da2ea" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302898 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302909 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.303190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.302917 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dff7e1-48b6-44b9-a59e-5f409d535387" containerName="cleanup" Nov 03 19:00:26.308695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.308675 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.358976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.358938 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl"] Nov 03 19:00:26.362122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.362090 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"distinct-2-w2-32a59b-kserve-self-signed-certs\"" Nov 03 19:00:26.449573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.449530 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-model-cache\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.467262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.467230 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-home\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.467576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.467553 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8wc\" (UniqueName: \"kubernetes.io/projected/dbf27b75-38f1-4318-b4f3-4101c02865f5-kube-api-access-rt8wc\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.467812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.467786 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-tmp-dir\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.468066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.468050 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf27b75-38f1-4318-b4f3-4101c02865f5-tls-certs\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.469283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.469262 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-dshm\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570085 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf27b75-38f1-4318-b4f3-4101c02865f5-tls-certs\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570137 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-dshm\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570193 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-model-cache\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570274 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-home\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570293 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8wc\" (UniqueName: \"kubernetes.io/projected/dbf27b75-38f1-4318-b4f3-4101c02865f5-kube-api-access-rt8wc\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570329 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-tmp-dir\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570719 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-model-cache\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.570841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.570810 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-home\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.571068 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.571045 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-tmp-dir\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.572949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.572926 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf27b75-38f1-4318-b4f3-4101c02865f5-tls-certs\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.573032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.572991 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-dshm\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.633652 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.633616 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8wc\" (UniqueName: \"kubernetes.io/projected/dbf27b75-38f1-4318-b4f3-4101c02865f5-kube-api-access-rt8wc\") pod \"distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:26.636841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.636816 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:00:26.922592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:26.922508 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:27.311182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:27.311152 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl"] Nov 03 19:00:27.645404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:27.645307 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" event={"ID":"dbf27b75-38f1-4318-b4f3-4101c02865f5","Type":"ContainerStarted","Data":"9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5"} Nov 03 19:00:27.645404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:27.645359 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" event={"ID":"dbf27b75-38f1-4318-b4f3-4101c02865f5","Type":"ContainerStarted","Data":"c5ce6e4f1e08abb6c389bdec751b946c8e7adfb49eede83b4e98cb690a28dbea"} Nov 03 19:00:27.767938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:27.767886 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podStartSLOduration=1.767871635 podStartE2EDuration="1.767871635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:27.764154486 +0000 UTC m=+1357.247150401" watchObservedRunningTime="2026-11-03 19:00:27.767871635 +0000 UTC m=+1357.250867550" Nov 03 19:00:27.926958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:27.926861 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:32.278234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.278199 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts"] Nov 03 19:00:32.279155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.279136 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:32.279204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.279158 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:32.279368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.279356 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a86af3-b752-4ba3-99b2-ceb6aedd9c74" containerName="cleanup" Nov 03 19:00:32.279403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.279374 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f48414-5c98-4ee8-9e7f-45cd3f1db6a8" containerName="cleanup" Nov 03 19:00:32.279403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.279386 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7588e6c6-b872-4193-8de8-990012b2e689" containerName="cleanup" Nov 03 19:00:32.284107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.284089 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.335546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.335513 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts"] Nov 03 19:00:32.379987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.379918 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:00:32.379987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.379980 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:00:32.380218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.379992 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:00:32.380218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.380009 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:00:32.455886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.455856 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.456084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.456050 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrxc\" (UniqueName: \"kubernetes.io/projected/bc7f2364-589b-49da-8ef1-7df46cbe4a92-kube-api-access-xgrxc\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.456149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.456115 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.557054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.556971 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrxc\" (UniqueName: \"kubernetes.io/projected/bc7f2364-589b-49da-8ef1-7df46cbe4a92-kube-api-access-xgrxc\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.557054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.557034 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.557260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.557077 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.557787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.557755 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.559637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.559611 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.619674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.619645 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrxc\" (UniqueName: \"kubernetes.io/projected/bc7f2364-589b-49da-8ef1-7df46cbe4a92-kube-api-access-xgrxc\") pod \"maas-api-revoke-sub-c8ed7bc1-cnjts\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:32.793911 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:00:32.793878 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:00:32.899822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:32.899747 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:33.280718 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:33.280655 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7f2364_589b_49da_8ef1_7df46cbe4a92.slice/crio-1fe64bb39d18d39a633dd5091a32fb9c49f28a13cf71ebdf2b405f44aa55be3a WatchSource:0}: Error finding container 1fe64bb39d18d39a633dd5091a32fb9c49f28a13cf71ebdf2b405f44aa55be3a: Status 404 returned error can't find the container with id 1fe64bb39d18d39a633dd5091a32fb9c49f28a13cf71ebdf2b405f44aa55be3a Nov 03 19:00:33.281389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:33.281361 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts"] Nov 03 19:00:33.697485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:33.697444 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" event={"ID":"bc7f2364-589b-49da-8ef1-7df46cbe4a92","Type":"ContainerStarted","Data":"87aff9994e7f8dc7daa0ed757061666aa5fc7602e7fe5f3f18de2bd02165da3c"} Nov 03 19:00:33.697687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:33.697507 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" event={"ID":"bc7f2364-589b-49da-8ef1-7df46cbe4a92","Type":"ContainerStarted","Data":"1fe64bb39d18d39a633dd5091a32fb9c49f28a13cf71ebdf2b405f44aa55be3a"} Nov 03 19:00:33.818931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:33.818806 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" podStartSLOduration=1.818788428 podStartE2EDuration="1.818788428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:33.815463981 +0000 UTC m=+1363.298459922" watchObservedRunningTime="2026-11-03 19:00:33.818788428 +0000 UTC m=+1363.301784341" Nov 03 19:00:34.709180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:34.709145 4948 generic.go:356] "Generic (PLEG): container finished" podID="bc7f2364-589b-49da-8ef1-7df46cbe4a92" containerID="87aff9994e7f8dc7daa0ed757061666aa5fc7602e7fe5f3f18de2bd02165da3c" exitCode=0 Nov 03 19:00:34.709677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:34.709235 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" event={"ID":"bc7f2364-589b-49da-8ef1-7df46cbe4a92","Type":"ContainerDied","Data":"87aff9994e7f8dc7daa0ed757061666aa5fc7602e7fe5f3f18de2bd02165da3c"} Nov 03 19:00:35.911059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:35.911032 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:35.999221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:35.999137 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-xks-ca\") pod \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " Nov 03 19:00:35.999384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:35.999222 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-openshift-ca\") pod \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " Nov 03 19:00:35.999384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:35.999354 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7f2364-589b-49da-8ef1-7df46cbe4a92-kube-api-access-xgrxc\" (UniqueName: \"kubernetes.io/projected/bc7f2364-589b-49da-8ef1-7df46cbe4a92-kube-api-access-xgrxc\") pod \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\" (UID: \"bc7f2364-589b-49da-8ef1-7df46cbe4a92\") " Nov 03 19:00:35.999634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:35.999609 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-openshift-ca" pod "bc7f2364-589b-49da-8ef1-7df46cbe4a92" (UID: "bc7f2364-589b-49da-8ef1-7df46cbe4a92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:35.999858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:35.999841 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:36.001340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.001315 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-xks-ca" pod "bc7f2364-589b-49da-8ef1-7df46cbe4a92" (UID: "bc7f2364-589b-49da-8ef1-7df46cbe4a92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:00:36.001460 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.001439 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7f2364-589b-49da-8ef1-7df46cbe4a92-kube-api-access-xgrxc" pod "bc7f2364-589b-49da-8ef1-7df46cbe4a92" (UID: "bc7f2364-589b-49da-8ef1-7df46cbe4a92"). InnerVolumeSpecName "kube-api-access-xgrxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:36.091757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.091723 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s"] Nov 03 19:00:36.099238 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.099211 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44b68e20-zf77s"] Nov 03 19:00:36.101302 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.101283 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrxc\" (UniqueName: \"kubernetes.io/projected/bc7f2364-589b-49da-8ef1-7df46cbe4a92-kube-api-access-xgrxc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:36.101364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.101307 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f2364-589b-49da-8ef1-7df46cbe4a92-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:36.730098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.730065 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" Nov 03 19:00:36.730280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.730095 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts" event={"ID":"bc7f2364-589b-49da-8ef1-7df46cbe4a92","Type":"ContainerDied","Data":"1fe64bb39d18d39a633dd5091a32fb9c49f28a13cf71ebdf2b405f44aa55be3a"} Nov 03 19:00:36.730280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.730169 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fe64bb39d18d39a633dd5091a32fb9c49f28a13cf71ebdf2b405f44aa55be3a" Nov 03 19:00:36.753990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.753955 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67a5a70-3f6d-4515-8504-0d908883fcdb" path="/var/lib/kubelet/pods/c67a5a70-3f6d-4515-8504-0d908883fcdb/volumes" Nov 03 19:00:36.923567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.923538 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:36.935876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:36.935850 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:37.751817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:37.751785 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:00:40.064582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:40.064545 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6"] Nov 03 19:00:40.115325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:40.115286 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8be2364a-wmrx6"] Nov 03 19:00:40.757713 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:40.757666 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f07be9-a414-4370-b5bc-3e738d78fcbd" path="/var/lib/kubelet/pods/a4f07be9-a414-4370-b5bc-3e738d78fcbd/volumes" Nov 03 19:00:46.633841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:46.633790 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:00:46.870847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:46.870801 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:00:47.026399 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:47.025824 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:00:48.389950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.389918 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9"] Nov 03 19:00:48.390690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.390674 4948 cpu_manager.go:414] "containerMap: removing container" podUID="bc7f2364-589b-49da-8ef1-7df46cbe4a92" containerName="revoke-keys" Nov 03 19:00:48.390766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.390690 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7f2364-589b-49da-8ef1-7df46cbe4a92" containerName="revoke-keys" Nov 03 19:00:48.390810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.390800 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7f2364-589b-49da-8ef1-7df46cbe4a92" containerName="revoke-keys" Nov 03 19:00:48.394797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.394779 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.447779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.447479 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.447779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.447656 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.448016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.447854 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854kp\" (UniqueName: \"kubernetes.io/projected/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-kube-api-access-854kp\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.461148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.450222 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9"] Nov 03 19:00:48.494602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.494559 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:00:48.496937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.496907 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:00:48.497093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.496955 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:00:48.500057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.500031 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv"] Nov 03 19:00:48.509013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.508991 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.548691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.548661 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.548853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.548715 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.548853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.548843 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.548978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.548888 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z749f\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-kube-api-access-z749f\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.548978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.548927 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.548978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.548948 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.549136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.549044 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.549136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.549098 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854kp\" (UniqueName: \"kubernetes.io/projected/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-kube-api-access-854kp\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.549234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.549136 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6637992b-c0a2-4488-b76a-a03ac96768d7-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.549333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.549307 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.549398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.549380 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.549449 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.549414 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.549633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.549610 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.552575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.552553 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv"] Nov 03 19:00:48.565083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.565061 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.567321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.567285 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:00:48.613372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.613339 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854kp\" (UniqueName: \"kubernetes.io/projected/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-kube-api-access-854kp\") pod \"maas-api-revoke-sub-380f942e-gqxs9\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.650663 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.650602 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.650663 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.650639 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.650881 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.650687 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.650881 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.650798 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6637992b-c0a2-4488-b76a-a03ac96768d7-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.650985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.650880 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.650985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.650949 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.650986 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.651016 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.651107 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.651127 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z749f\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-kube-api-access-z749f\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.651405 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.651599 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6637992b-c0a2-4488-b76a-a03ac96768d7-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.651722 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.651743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.651730 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.653067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.653040 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.653987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.653967 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.709541 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.709513 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:48.713053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.713025 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.713195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.713052 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z749f\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-kube-api-access-z749f\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.823387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.823274 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:48.905377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.905347 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb"] Nov 03 19:00:48.913003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.912256 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.961336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.961297 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb"] Nov 03 19:00:48.962273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962248 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962321 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962516 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962496 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962527 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqk52\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-kube-api-access-pqk52\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962556 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d283713d-1736-4544-92a8-618ffc320af5-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962612 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d283713d-1736-4544-92a8-618ffc320af5-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962666 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962721 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:48.962822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:48.962783 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064505 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d283713d-1736-4544-92a8-618ffc320af5-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064568 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064607 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064626 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064740 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064782 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064873 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064899 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqk52\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-kube-api-access-pqk52\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.064990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.064929 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d283713d-1736-4544-92a8-618ffc320af5-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.065242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.065078 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.065242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.065183 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.065368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.065345 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d283713d-1736-4544-92a8-618ffc320af5-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.065624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.065597 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.065728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.065625 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.067312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.067293 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.067441 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.067423 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d283713d-1736-4544-92a8-618ffc320af5-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.097496 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:49.097472 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0851f0a5_62e9_479b_a7b2_7ae682b23ca2.slice/crio-44450b92e4f92f5217818581e71d4fb0d799612a7083d630a3ad8bb216d87843 WatchSource:0}: Error finding container 44450b92e4f92f5217818581e71d4fb0d799612a7083d630a3ad8bb216d87843: Status 404 returned error can't find the container with id 44450b92e4f92f5217818581e71d4fb0d799612a7083d630a3ad8bb216d87843 Nov 03 19:00:49.097973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.097910 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9"] Nov 03 19:00:49.127643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.127623 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.127840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.127819 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqk52\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-kube-api-access-pqk52\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.196839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.196812 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d"] Nov 03 19:00:49.202836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.202816 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.226771 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:49.226742 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6637992b_c0a2_4488_b76a_a03ac96768d7.slice/crio-f8480b8650352fce76e1ad49977a21e9a59fa6b3d479cf3d3d548995dd55e5b5 WatchSource:0}: Error finding container f8480b8650352fce76e1ad49977a21e9a59fa6b3d479cf3d3d548995dd55e5b5: Status 404 returned error can't find the container with id f8480b8650352fce76e1ad49977a21e9a59fa6b3d479cf3d3d548995dd55e5b5 Nov 03 19:00:49.228318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.228294 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv"] Nov 03 19:00:49.229601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.229567 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.229690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.229631 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.229690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.229666 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.230172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.230102 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:49.261712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.261670 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d"] Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270093 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270199 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270266 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270299 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270387 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270423 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270476 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz7dq\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-kube-api-access-vz7dq\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270518 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.270896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.270550 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651bbe89-2f02-462c-bc24-cf5a69fbd101-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.383234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.383189 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.387437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.387141 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.388166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.388145 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.388401 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.388384 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.388950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.388930 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.389446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.389321 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.389859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.389835 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz7dq\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-kube-api-access-vz7dq\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.390029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.390013 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.390904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.390870 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651bbe89-2f02-462c-bc24-cf5a69fbd101-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.391965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.390798 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.391965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.391806 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651bbe89-2f02-462c-bc24-cf5a69fbd101-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.392260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.392242 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.392750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.392617 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.392919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.392895 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.397631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.397386 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.397631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.397596 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.452033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.451991 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz7dq\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-kube-api-access-vz7dq\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.455679 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.455634 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.517150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.517119 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:49.656239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.656074 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb"] Nov 03 19:00:49.663445 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.662021 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.663445 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.662125 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.663445 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.662177 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.925861 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:49.925825 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651bbe89_2f02_462c_bc24_cf5a69fbd101.slice/crio-b150d107528b1636e14e8d7c09e23b26f8394f707d29ee2e067fd9707bd7582a WatchSource:0}: Error finding container b150d107528b1636e14e8d7c09e23b26f8394f707d29ee2e067fd9707bd7582a: Status 404 returned error can't find the container with id b150d107528b1636e14e8d7c09e23b26f8394f707d29ee2e067fd9707bd7582a Nov 03 19:00:49.926830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.926800 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d"] Nov 03 19:00:49.928489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.928452 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.928585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.928527 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:49.928585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:49.928562 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:00:50.043385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.043343 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" event={"ID":"d283713d-1736-4544-92a8-618ffc320af5","Type":"ContainerStarted","Data":"107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3"} Nov 03 19:00:50.043537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.043410 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" event={"ID":"d283713d-1736-4544-92a8-618ffc320af5","Type":"ContainerStarted","Data":"de259e1db36899ce147c1bbad487b527d37bc8ad668e130a59011f8364f8cccf"} Nov 03 19:00:50.045682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.045655 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" event={"ID":"651bbe89-2f02-462c-bc24-cf5a69fbd101","Type":"ContainerStarted","Data":"7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3"} Nov 03 19:00:50.046548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.045746 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" event={"ID":"651bbe89-2f02-462c-bc24-cf5a69fbd101","Type":"ContainerStarted","Data":"b150d107528b1636e14e8d7c09e23b26f8394f707d29ee2e067fd9707bd7582a"} Nov 03 19:00:50.048619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.048591 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" event={"ID":"0851f0a5-62e9-479b-a7b2-7ae682b23ca2","Type":"ContainerStarted","Data":"e5892109a42ca92d4bd54c8c8bb213b62c270577a4fbc271e9ffff82fad4992a"} Nov 03 19:00:50.048769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.048638 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" event={"ID":"0851f0a5-62e9-479b-a7b2-7ae682b23ca2","Type":"ContainerStarted","Data":"44450b92e4f92f5217818581e71d4fb0d799612a7083d630a3ad8bb216d87843"} Nov 03 19:00:50.050892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.050852 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" event={"ID":"6637992b-c0a2-4488-b76a-a03ac96768d7","Type":"ContainerStarted","Data":"31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510"} Nov 03 19:00:50.051018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.050902 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" event={"ID":"6637992b-c0a2-4488-b76a-a03ac96768d7","Type":"ContainerStarted","Data":"f8480b8650352fce76e1ad49977a21e9a59fa6b3d479cf3d3d548995dd55e5b5"} Nov 03 19:00:50.180877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.180816 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podStartSLOduration=2.180791574 podStartE2EDuration="2.180791574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.16877282 +0000 UTC m=+1379.651768736" watchObservedRunningTime="2026-11-03 19:00:50.180791574 +0000 UTC m=+1379.663787487" Nov 03 19:00:50.230686 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.230651 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:50.231788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.231760 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:50.231868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.231816 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:50.289947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.289794 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" podStartSLOduration=2.289772471 podStartE2EDuration="2.289772471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.286749981 +0000 UTC m=+1379.769745896" watchObservedRunningTime="2026-11-03 19:00:50.289772471 +0000 UTC m=+1379.772768387" Nov 03 19:00:50.427529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.427467 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podStartSLOduration=2.427453528 podStartE2EDuration="2.427453528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.426218024 +0000 UTC m=+1379.909213931" watchObservedRunningTime="2026-11-03 19:00:50.427453528 +0000 UTC m=+1379.910449442" Nov 03 19:00:50.519162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.519129 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:50.586723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.582479 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podStartSLOduration=1.582450074 podStartE2EDuration="1.582450074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.548204334 +0000 UTC m=+1380.031200250" watchObservedRunningTime="2026-11-03 19:00:50.582450074 +0000 UTC m=+1380.065445992" Nov 03 19:00:50.830147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.830059 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:50.833135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.833086 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:00:50.833415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:50.833375 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 19:00:51.205786 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.123098 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb"] Nov 03 19:00:51.301301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.288152 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.317774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.314228 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:51.317774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.314288 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:51.467954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.467868 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.496940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.496898 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fstw\" (UniqueName: \"kubernetes.io/projected/ccb86547-af7a-419e-b2e0-8515a3ef3730-kube-api-access-4fstw\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.497240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.497219 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.538725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.499216 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8"] Nov 03 19:00:51.640937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.600225 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:00:51.640937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.601241 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fstw\" (UniqueName: \"kubernetes.io/projected/ccb86547-af7a-419e-b2e0-8515a3ef3730-kube-api-access-4fstw\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.640937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.617668 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.640937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.618023 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.648144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.648099 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.650123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.650102 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4f6a24-n5vr8"] Nov 03 19:00:51.671676 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.671635 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:00:51.703213 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.703177 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.716066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.716023 4948 generic.go:356] "Generic (PLEG): container finished" podID="0851f0a5-62e9-479b-a7b2-7ae682b23ca2" containerID="e5892109a42ca92d4bd54c8c8bb213b62c270577a4fbc271e9ffff82fad4992a" exitCode=0 Nov 03 19:00:51.759508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.728380 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb"] Nov 03 19:00:51.759720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.759677 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" event={"ID":"0851f0a5-62e9-479b-a7b2-7ae682b23ca2","Type":"ContainerDied","Data":"e5892109a42ca92d4bd54c8c8bb213b62c270577a4fbc271e9ffff82fad4992a"} Nov 03 19:00:51.821725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.795446 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fstw\" (UniqueName: \"kubernetes.io/projected/ccb86547-af7a-419e-b2e0-8515a3ef3730-kube-api-access-4fstw\") pod \"maas-api-revoke-sub-621c6a45-6cpzb\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:51.830726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.824511 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:00:51.830726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.824568 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 19:00:51.920118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:51.920078 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:52.250224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.250187 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:52.250523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.250489 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:52.486742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.473461 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:00:52.524600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.524344 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:00:52.524600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.524412 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:00:52.813800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.803019 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a777292e-b384-4354-9b75-54117e215b5f" path="/var/lib/kubelet/pods/a777292e-b384-4354-9b75-54117e215b5f/volumes" Nov 03 19:00:52.824888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.824307 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:00:52.824888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:52.824383 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 19:00:53.285727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.280283 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:53.285727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.280349 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:53.586854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.586773 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:00:53.587776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.587344 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:00:53.624723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.617839 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:00:53.633720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.628641 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:00:53.832592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.832558 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb"] Nov 03 19:00:53.857792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.854107 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:00:53.857792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.854185 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 19:00:53.875531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:53.875491 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:00:54.065722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.053893 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:54.095529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.095484 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" event={"ID":"ccb86547-af7a-419e-b2e0-8515a3ef3730","Type":"ContainerStarted","Data":"e855be85d9c75356773621f3c3cd8f77717af7fa24e1606003a5526be33e8aad"} Nov 03 19:00:54.099627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.099598 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" event={"ID":"0851f0a5-62e9-479b-a7b2-7ae682b23ca2","Type":"ContainerDied","Data":"44450b92e4f92f5217818581e71d4fb0d799612a7083d630a3ad8bb216d87843"} Nov 03 19:00:54.103947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.103927 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44450b92e4f92f5217818581e71d4fb0d799612a7083d630a3ad8bb216d87843" Nov 03 19:00:54.104095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.103876 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9" Nov 03 19:00:54.148306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.148204 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-openshift-ca\") pod \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " Nov 03 19:00:54.148657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.148632 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-xks-ca\") pod \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " Nov 03 19:00:54.148920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.148900 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-kube-api-access-854kp\" (UniqueName: \"kubernetes.io/projected/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-kube-api-access-854kp\") pod \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\" (UID: \"0851f0a5-62e9-479b-a7b2-7ae682b23ca2\") " Nov 03 19:00:54.210953 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.210889 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-xks-ca" pod "0851f0a5-62e9-479b-a7b2-7ae682b23ca2" (UID: "0851f0a5-62e9-479b-a7b2-7ae682b23ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:00:54.224088 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.224045 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-kube-api-access-854kp" pod "0851f0a5-62e9-479b-a7b2-7ae682b23ca2" (UID: "0851f0a5-62e9-479b-a7b2-7ae682b23ca2"). InnerVolumeSpecName "kube-api-access-854kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:54.304771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.297046 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:54.306149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.306121 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854kp\" (UniqueName: \"kubernetes.io/projected/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-kube-api-access-854kp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:54.306574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.306545 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-openshift-ca" pod "0851f0a5-62e9-479b-a7b2-7ae682b23ca2" (UID: "0851f0a5-62e9-479b-a7b2-7ae682b23ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:54.306854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.306835 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:54.307029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.307005 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:54.407496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.407464 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0851f0a5-62e9-479b-a7b2-7ae682b23ca2-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:54.527736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.524155 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:00:54.527736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.524215 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:00:54.838359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.838322 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:00:54.842997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:54.842965 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 19:00:55.150728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:55.150489 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" event={"ID":"ccb86547-af7a-419e-b2e0-8515a3ef3730","Type":"ContainerStarted","Data":"d6423b1b974dff960315f029b7a2e1a65c98756bd2ba9afabaa7c4e46676ba56"} Nov 03 19:00:55.257877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:55.254794 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:55.257877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:55.254858 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:55.522255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:55.522182 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:00:55.522612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:55.522492 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:00:55.824651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:55.824134 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:00:55.824651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:55.824198 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 19:00:56.200725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.195622 4948 generic.go:356] "Generic (PLEG): container finished" podID="ccb86547-af7a-419e-b2e0-8515a3ef3730" containerID="d6423b1b974dff960315f029b7a2e1a65c98756bd2ba9afabaa7c4e46676ba56" exitCode=0 Nov 03 19:00:56.200725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.195921 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" event={"ID":"ccb86547-af7a-419e-b2e0-8515a3ef3730","Type":"ContainerDied","Data":"d6423b1b974dff960315f029b7a2e1a65c98756bd2ba9afabaa7c4e46676ba56"} Nov 03 19:00:56.232723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.232095 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:56.232723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.232155 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:56.517873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.517782 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:00:56.517873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.517849 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:00:56.626334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.626297 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx"] Nov 03 19:00:56.627637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.627594 4948 cpu_manager.go:414] "containerMap: removing container" podUID="0851f0a5-62e9-479b-a7b2-7ae682b23ca2" containerName="revoke-keys" Nov 03 19:00:56.627867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.627848 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0851f0a5-62e9-479b-a7b2-7ae682b23ca2" containerName="revoke-keys" Nov 03 19:00:56.720271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.646639 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="0851f0a5-62e9-479b-a7b2-7ae682b23ca2" containerName="revoke-keys" Nov 03 19:00:56.723739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.723575 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx"] Nov 03 19:00:56.725087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.725062 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.824393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.824299 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:00:56.824393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.824359 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 19:00:56.835268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.835230 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.835423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.835284 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwpm\" (UniqueName: \"kubernetes.io/projected/c6668c8b-1fe7-441a-a599-f748e448e518-kube-api-access-mfwpm\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.835472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.835431 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.937259 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.937209 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.937551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.937279 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwpm\" (UniqueName: \"kubernetes.io/projected/c6668c8b-1fe7-441a-a599-f748e448e518-kube-api-access-mfwpm\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.937551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.937330 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.938312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.938283 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:56.941117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:56.941089 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:57.000719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.000510 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwpm\" (UniqueName: \"kubernetes.io/projected/c6668c8b-1fe7-441a-a599-f748e448e518-kube-api-access-mfwpm\") pod \"maas-api-revoke-sub-4f1e9946-b5shx\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:57.049330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.049289 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:00:57.232292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.231858 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:00:57.232292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.231917 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:00:57.515992 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:00:57.515957 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6668c8b_1fe7_441a_a599_f748e448e518.slice/crio-59241ae73a4e826134c796c4ffec6bdad954359e104d12120a2273b7a8c67e0b WatchSource:0}: Error finding container 59241ae73a4e826134c796c4ffec6bdad954359e104d12120a2273b7a8c67e0b: Status 404 returned error can't find the container with id 59241ae73a4e826134c796c4ffec6bdad954359e104d12120a2273b7a8c67e0b Nov 03 19:00:57.520513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.520121 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 19:00:57.520513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.520170 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 19:00:57.531362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.531332 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx"] Nov 03 19:00:57.541221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.540866 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:57.610720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.610628 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-openshift-ca\") pod \"ccb86547-af7a-419e-b2e0-8515a3ef3730\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " Nov 03 19:00:57.610720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.610724 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb86547-af7a-419e-b2e0-8515a3ef3730-kube-api-access-4fstw\" (UniqueName: \"kubernetes.io/projected/ccb86547-af7a-419e-b2e0-8515a3ef3730-kube-api-access-4fstw\") pod \"ccb86547-af7a-419e-b2e0-8515a3ef3730\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " Nov 03 19:00:57.610976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.610751 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-xks-ca\") pod \"ccb86547-af7a-419e-b2e0-8515a3ef3730\" (UID: \"ccb86547-af7a-419e-b2e0-8515a3ef3730\") " Nov 03 19:00:57.614730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.612287 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-openshift-ca" pod "ccb86547-af7a-419e-b2e0-8515a3ef3730" (UID: "ccb86547-af7a-419e-b2e0-8515a3ef3730"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:57.617732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.617672 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-xks-ca" pod "ccb86547-af7a-419e-b2e0-8515a3ef3730" (UID: "ccb86547-af7a-419e-b2e0-8515a3ef3730"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:00:57.619044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.619018 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb86547-af7a-419e-b2e0-8515a3ef3730-kube-api-access-4fstw" pod "ccb86547-af7a-419e-b2e0-8515a3ef3730" (UID: "ccb86547-af7a-419e-b2e0-8515a3ef3730"). InnerVolumeSpecName "kube-api-access-4fstw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:57.721713 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.721663 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:57.721988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.721963 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fstw\" (UniqueName: \"kubernetes.io/projected/ccb86547-af7a-419e-b2e0-8515a3ef3730-kube-api-access-4fstw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:57.722133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:57.722120 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb86547-af7a-419e-b2e0-8515a3ef3730-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:58.124761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.122262 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:58.124761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.123849 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:58.125796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.125767 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:00:58.241959 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.241911 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" event={"ID":"c6668c8b-1fe7-441a-a599-f748e448e518","Type":"ContainerStarted","Data":"2dfdbe69cf66ebd677f56e03fae5881a09259b58ac0ff06411ee2d4d9a27b325"} Nov 03 19:00:58.242475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.241984 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" event={"ID":"c6668c8b-1fe7-441a-a599-f748e448e518","Type":"ContainerStarted","Data":"59241ae73a4e826134c796c4ffec6bdad954359e104d12120a2273b7a8c67e0b"} Nov 03 19:00:58.245496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.244636 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" Nov 03 19:00:58.246658 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.246630 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb" event={"ID":"ccb86547-af7a-419e-b2e0-8515a3ef3730","Type":"ContainerDied","Data":"e855be85d9c75356773621f3c3cd8f77717af7fa24e1606003a5526be33e8aad"} Nov 03 19:00:58.246853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.246835 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e855be85d9c75356773621f3c3cd8f77717af7fa24e1606003a5526be33e8aad" Nov 03 19:00:58.367048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.366948 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" podStartSLOduration=2.366932221 podStartE2EDuration="2.366932221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:58.366215049 +0000 UTC m=+1387.849210965" watchObservedRunningTime="2026-11-03 19:00:58.366932221 +0000 UTC m=+1387.849928135" Nov 03 19:00:58.431105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.430925 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:58.432554 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.431405 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:58.433493 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.433471 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:00:58.524307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.524275 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:58.524723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.524680 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:58.525692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:58.525669 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:00:59.264844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:59.264808 4948 generic.go:356] "Generic (PLEG): container finished" podID="c6668c8b-1fe7-441a-a599-f748e448e518" containerID="2dfdbe69cf66ebd677f56e03fae5881a09259b58ac0ff06411ee2d4d9a27b325" exitCode=0 Nov 03 19:00:59.265288 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:00:59.264904 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" event={"ID":"c6668c8b-1fe7-441a-a599-f748e448e518","Type":"ContainerDied","Data":"2dfdbe69cf66ebd677f56e03fae5881a09259b58ac0ff06411ee2d4d9a27b325"} Nov 03 19:01:00.053284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.053249 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b"] Nov 03 19:01:00.103683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.103655 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad344f19-tk42b"] Nov 03 19:01:00.422596 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.422571 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:01:00.502570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.502528 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-xks-ca\") pod \"c6668c8b-1fe7-441a-a599-f748e448e518\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " Nov 03 19:01:00.502788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.502578 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-openshift-ca\") pod \"c6668c8b-1fe7-441a-a599-f748e448e518\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " Nov 03 19:01:00.502788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.502629 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6668c8b-1fe7-441a-a599-f748e448e518-kube-api-access-mfwpm\" (UniqueName: \"kubernetes.io/projected/c6668c8b-1fe7-441a-a599-f748e448e518-kube-api-access-mfwpm\") pod \"c6668c8b-1fe7-441a-a599-f748e448e518\" (UID: \"c6668c8b-1fe7-441a-a599-f748e448e518\") " Nov 03 19:01:00.503069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.503038 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-openshift-ca" pod "c6668c8b-1fe7-441a-a599-f748e448e518" (UID: "c6668c8b-1fe7-441a-a599-f748e448e518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:00.504655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.504632 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-xks-ca" pod "c6668c8b-1fe7-441a-a599-f748e448e518" (UID: "c6668c8b-1fe7-441a-a599-f748e448e518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:00.504816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.504794 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6668c8b-1fe7-441a-a599-f748e448e518-kube-api-access-mfwpm" pod "c6668c8b-1fe7-441a-a599-f748e448e518" (UID: "c6668c8b-1fe7-441a-a599-f748e448e518"). InnerVolumeSpecName "kube-api-access-mfwpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:00.603813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.603738 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:00.603813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.603768 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6668c8b-1fe7-441a-a599-f748e448e518-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:00.603813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.603778 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwpm\" (UniqueName: \"kubernetes.io/projected/c6668c8b-1fe7-441a-a599-f748e448e518-kube-api-access-mfwpm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:00.756248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:00.756175 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ff7716-3db1-42bf-9bfa-b8f9e6ccf521" path="/var/lib/kubelet/pods/84ff7716-3db1-42bf-9bfa-b8f9e6ccf521/volumes" Nov 03 19:01:01.343744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:01.337446 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" event={"ID":"c6668c8b-1fe7-441a-a599-f748e448e518","Type":"ContainerDied","Data":"59241ae73a4e826134c796c4ffec6bdad954359e104d12120a2273b7a8c67e0b"} Nov 03 19:01:01.343744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:01.337509 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59241ae73a4e826134c796c4ffec6bdad954359e104d12120a2273b7a8c67e0b" Nov 03 19:01:01.343744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:01.337577 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx" Nov 03 19:01:02.794356 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:01:02.794318 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:01:04.107559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.107517 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk"] Nov 03 19:01:04.108620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.108596 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ccb86547-af7a-419e-b2e0-8515a3ef3730" containerName="revoke-keys" Nov 03 19:01:04.108695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.108623 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb86547-af7a-419e-b2e0-8515a3ef3730" containerName="revoke-keys" Nov 03 19:01:04.108695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.108644 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c6668c8b-1fe7-441a-a599-f748e448e518" containerName="revoke-keys" Nov 03 19:01:04.108695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.108652 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6668c8b-1fe7-441a-a599-f748e448e518" containerName="revoke-keys" Nov 03 19:01:04.108891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.108853 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb86547-af7a-419e-b2e0-8515a3ef3730" containerName="revoke-keys" Nov 03 19:01:04.108891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.108868 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6668c8b-1fe7-441a-a599-f748e448e518" containerName="revoke-keys" Nov 03 19:01:04.114602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.114579 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.162546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.162515 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk"] Nov 03 19:01:04.167255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.167222 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.167412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.167390 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.167493 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.167429 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28xx\" (UniqueName: \"kubernetes.io/projected/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-kube-api-access-l28xx\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.207888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.207860 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:01:04.208054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.207950 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:01:04.208054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.207974 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:01:04.219398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.219376 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:01:04.269136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.269104 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.269267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.269149 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28xx\" (UniqueName: \"kubernetes.io/projected/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-kube-api-access-l28xx\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.269323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.269306 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.270028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.269912 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.271967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.271940 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.331526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.331500 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28xx\" (UniqueName: \"kubernetes.io/projected/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-kube-api-access-l28xx\") pod \"maas-api-revoke-sub-a8b3898a-n6khk\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.430637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.430554 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:04.825558 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:04.825522 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047fa32c_c3e4_4fde_b5ea_37baaf608c3d.slice/crio-f9a4a84f47d74846176d9f57a8bcfb46717cf563a8408a2eca14872a658d6794 WatchSource:0}: Error finding container f9a4a84f47d74846176d9f57a8bcfb46717cf563a8408a2eca14872a658d6794: Status 404 returned error can't find the container with id f9a4a84f47d74846176d9f57a8bcfb46717cf563a8408a2eca14872a658d6794 Nov 03 19:01:04.826567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:04.826547 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk"] Nov 03 19:01:05.376964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:05.376877 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" event={"ID":"047fa32c-c3e4-4fde-b5ea-37baaf608c3d","Type":"ContainerStarted","Data":"b214cc12af718b0001be62344b37ff9968663a5cf10151311156cda06e2da774"} Nov 03 19:01:05.376964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:05.376940 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" event={"ID":"047fa32c-c3e4-4fde-b5ea-37baaf608c3d","Type":"ContainerStarted","Data":"f9a4a84f47d74846176d9f57a8bcfb46717cf563a8408a2eca14872a658d6794"} Nov 03 19:01:05.498214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:05.498160 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" podStartSLOduration=1.498145872 podStartE2EDuration="1.498145872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:05.495119198 +0000 UTC m=+1394.978115114" watchObservedRunningTime="2026-11-03 19:01:05.498145872 +0000 UTC m=+1394.981141785" Nov 03 19:01:05.890559 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:01:05.890524 4948 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047fa32c_c3e4_4fde_b5ea_37baaf608c3d.slice/crio-conmon-b214cc12af718b0001be62344b37ff9968663a5cf10151311156cda06e2da774.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:01:06.386128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:06.386092 4948 generic.go:356] "Generic (PLEG): container finished" podID="047fa32c-c3e4-4fde-b5ea-37baaf608c3d" containerID="b214cc12af718b0001be62344b37ff9968663a5cf10151311156cda06e2da774" exitCode=0 Nov 03 19:01:06.386550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:06.386183 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" event={"ID":"047fa32c-c3e4-4fde-b5ea-37baaf608c3d","Type":"ContainerDied","Data":"b214cc12af718b0001be62344b37ff9968663a5cf10151311156cda06e2da774"} Nov 03 19:01:07.563552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.563524 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:07.614601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.614568 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-openshift-ca\") pod \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " Nov 03 19:01:07.614803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.614776 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-kube-api-access-l28xx\" (UniqueName: \"kubernetes.io/projected/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-kube-api-access-l28xx\") pod \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " Nov 03 19:01:07.614946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.614926 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-xks-ca\") pod \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\" (UID: \"047fa32c-c3e4-4fde-b5ea-37baaf608c3d\") " Nov 03 19:01:07.615055 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.615025 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-openshift-ca" pod "047fa32c-c3e4-4fde-b5ea-37baaf608c3d" (UID: "047fa32c-c3e4-4fde-b5ea-37baaf608c3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:07.615493 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.615466 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:07.617367 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.617331 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-xks-ca" pod "047fa32c-c3e4-4fde-b5ea-37baaf608c3d" (UID: "047fa32c-c3e4-4fde-b5ea-37baaf608c3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:07.623418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.623394 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-kube-api-access-l28xx" pod "047fa32c-c3e4-4fde-b5ea-37baaf608c3d" (UID: "047fa32c-c3e4-4fde-b5ea-37baaf608c3d"). InnerVolumeSpecName "kube-api-access-l28xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:07.716872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.716840 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l28xx\" (UniqueName: \"kubernetes.io/projected/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-kube-api-access-l28xx\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:07.717028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:07.716876 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047fa32c-c3e4-4fde-b5ea-37baaf608c3d-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:08.406126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:08.406100 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" Nov 03 19:01:08.406306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:08.406136 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk" event={"ID":"047fa32c-c3e4-4fde-b5ea-37baaf608c3d","Type":"ContainerDied","Data":"f9a4a84f47d74846176d9f57a8bcfb46717cf563a8408a2eca14872a658d6794"} Nov 03 19:01:08.406306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:08.406186 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a4a84f47d74846176d9f57a8bcfb46717cf563a8408a2eca14872a658d6794" Nov 03 19:01:11.771328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.771289 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb"] Nov 03 19:01:11.772161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.772138 4948 cpu_manager.go:414] "containerMap: removing container" podUID="047fa32c-c3e4-4fde-b5ea-37baaf608c3d" containerName="revoke-keys" Nov 03 19:01:11.772161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.772161 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047fa32c-c3e4-4fde-b5ea-37baaf608c3d" containerName="revoke-keys" Nov 03 19:01:11.772380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.772365 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="047fa32c-c3e4-4fde-b5ea-37baaf608c3d" containerName="revoke-keys" Nov 03 19:01:11.776073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.776050 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.830097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.830066 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb"] Nov 03 19:01:11.873027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.872991 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zxg\" (UniqueName: \"kubernetes.io/projected/a3398b2e-0c7a-4261-8152-b730c6124acd-kube-api-access-l4zxg\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.873295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.873271 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.873404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.873328 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.875217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.875197 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:01:11.875318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.875231 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:01:11.875318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.875266 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:01:11.875318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.875304 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:01:11.974314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.974280 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zxg\" (UniqueName: \"kubernetes.io/projected/a3398b2e-0c7a-4261-8152-b730c6124acd-kube-api-access-l4zxg\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.974503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.974329 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.974503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.974363 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.975214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.975188 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:11.987103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:11.987077 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:12.036878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:12.036811 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zxg\" (UniqueName: \"kubernetes.io/projected/a3398b2e-0c7a-4261-8152-b730c6124acd-kube-api-access-l4zxg\") pod \"maas-api-revoke-sub-b81a9bd8-zz8nb\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:12.053330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:12.053301 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv"] Nov 03 19:01:12.058087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:12.058061 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbead74c-w66gv"] Nov 03 19:01:12.090437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:12.090401 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:12.474792 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:12.474757 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3398b2e_0c7a_4261_8152_b730c6124acd.slice/crio-14da0692a5cc9dfa08353d7b8afa57dbaf94706f8659500b3c59ed69cb5dfcc3 WatchSource:0}: Error finding container 14da0692a5cc9dfa08353d7b8afa57dbaf94706f8659500b3c59ed69cb5dfcc3: Status 404 returned error can't find the container with id 14da0692a5cc9dfa08353d7b8afa57dbaf94706f8659500b3c59ed69cb5dfcc3 Nov 03 19:01:12.477281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:12.477254 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb"] Nov 03 19:01:12.758993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:12.758961 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2980a7cb-b1fa-462d-b22c-40888df69c99" path="/var/lib/kubelet/pods/2980a7cb-b1fa-462d-b22c-40888df69c99/volumes" Nov 03 19:01:13.464471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:13.464429 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" event={"ID":"a3398b2e-0c7a-4261-8152-b730c6124acd","Type":"ContainerStarted","Data":"5f388d941950caf50e2c5d6960b886e04555dbe2d95799cd661d59d221cf4839"} Nov 03 19:01:13.464861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:13.464531 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" event={"ID":"a3398b2e-0c7a-4261-8152-b730c6124acd","Type":"ContainerStarted","Data":"14da0692a5cc9dfa08353d7b8afa57dbaf94706f8659500b3c59ed69cb5dfcc3"} Nov 03 19:01:14.480466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.480429 4948 generic.go:356] "Generic (PLEG): container finished" podID="a3398b2e-0c7a-4261-8152-b730c6124acd" containerID="5f388d941950caf50e2c5d6960b886e04555dbe2d95799cd661d59d221cf4839" exitCode=0 Nov 03 19:01:14.480919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.480518 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" event={"ID":"a3398b2e-0c7a-4261-8152-b730c6124acd","Type":"ContainerDied","Data":"5f388d941950caf50e2c5d6960b886e04555dbe2d95799cd661d59d221cf4839"} Nov 03 19:01:14.644576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.644553 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:14.811369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.811286 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-openshift-ca\") pod \"a3398b2e-0c7a-4261-8152-b730c6124acd\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " Nov 03 19:01:14.811605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.811446 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3398b2e-0c7a-4261-8152-b730c6124acd-kube-api-access-l4zxg\" (UniqueName: \"kubernetes.io/projected/a3398b2e-0c7a-4261-8152-b730c6124acd-kube-api-access-l4zxg\") pod \"a3398b2e-0c7a-4261-8152-b730c6124acd\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " Nov 03 19:01:14.811605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.811565 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-xks-ca\") pod \"a3398b2e-0c7a-4261-8152-b730c6124acd\" (UID: \"a3398b2e-0c7a-4261-8152-b730c6124acd\") " Nov 03 19:01:14.811971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.811886 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-openshift-ca" pod "a3398b2e-0c7a-4261-8152-b730c6124acd" (UID: "a3398b2e-0c7a-4261-8152-b730c6124acd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:14.812182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.812152 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:14.814448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.814424 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3398b2e-0c7a-4261-8152-b730c6124acd-kube-api-access-l4zxg" pod "a3398b2e-0c7a-4261-8152-b730c6124acd" (UID: "a3398b2e-0c7a-4261-8152-b730c6124acd"). InnerVolumeSpecName "kube-api-access-l4zxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:14.814550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.814420 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-xks-ca" pod "a3398b2e-0c7a-4261-8152-b730c6124acd" (UID: "a3398b2e-0c7a-4261-8152-b730c6124acd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:14.913435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.913395 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zxg\" (UniqueName: \"kubernetes.io/projected/a3398b2e-0c7a-4261-8152-b730c6124acd-kube-api-access-l4zxg\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:14.913435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:14.913433 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3398b2e-0c7a-4261-8152-b730c6124acd-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:15.492553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:15.492519 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" Nov 03 19:01:15.492950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:15.492560 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb" event={"ID":"a3398b2e-0c7a-4261-8152-b730c6124acd","Type":"ContainerDied","Data":"14da0692a5cc9dfa08353d7b8afa57dbaf94706f8659500b3c59ed69cb5dfcc3"} Nov 03 19:01:15.492950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:15.492620 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14da0692a5cc9dfa08353d7b8afa57dbaf94706f8659500b3c59ed69cb5dfcc3" Nov 03 19:01:19.474477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.474430 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4"] Nov 03 19:01:19.475440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.475417 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a3398b2e-0c7a-4261-8152-b730c6124acd" containerName="revoke-keys" Nov 03 19:01:19.475500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.475443 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3398b2e-0c7a-4261-8152-b730c6124acd" containerName="revoke-keys" Nov 03 19:01:19.475658 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.475645 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3398b2e-0c7a-4261-8152-b730c6124acd" containerName="revoke-keys" Nov 03 19:01:19.480214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.480190 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.529986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.529953 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4"] Nov 03 19:01:19.573454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.573418 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.573868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.573840 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.574086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.574060 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfcv\" (UniqueName: \"kubernetes.io/projected/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-kube-api-access-swfcv\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.580240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.580217 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:01:19.580354 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.580227 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:01:19.580354 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.580285 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:01:19.633984 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.633956 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:01:19.675859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.675828 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.676051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.675968 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.676051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.676000 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfcv\" (UniqueName: \"kubernetes.io/projected/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-kube-api-access-swfcv\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.676711 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.676673 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.678483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.678448 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.749383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.749300 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfcv\" (UniqueName: \"kubernetes.io/projected/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-kube-api-access-swfcv\") pod \"maas-api-revoke-sub-c31f13c8-swpw4\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:19.795873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:19.795832 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:20.041385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.041295 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6"] Nov 03 19:01:20.047222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.047197 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081094 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b567e2b-d032-4733-a6bb-6531f0d149a2-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081169 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081260 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66h9v\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-kube-api-access-66h9v\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081294 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081355 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081390 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081434 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081480 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.081646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.081547 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.160582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.160543 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6"] Nov 03 19:01:20.183239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183193 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b567e2b-d032-4733-a6bb-6531f0d149a2-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183320 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183382 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66h9v\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-kube-api-access-66h9v\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183422 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183468 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183519 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183597 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183643 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.183691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.183674 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.185293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.184758 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.185293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.185069 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.185293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.185209 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b567e2b-d032-4733-a6bb-6531f0d149a2-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.185556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.185514 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.185869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.185845 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.190331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.190301 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.208358 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:20.208330 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e8dbf6_c74b_4371_bc3d_0e0a6288522b.slice/crio-42e8fd9e8a10fcc294fb60c9450582f08935f406555ddc021ec233e604195c3c WatchSource:0}: Error finding container 42e8fd9e8a10fcc294fb60c9450582f08935f406555ddc021ec233e604195c3c: Status 404 returned error can't find the container with id 42e8fd9e8a10fcc294fb60c9450582f08935f406555ddc021ec233e604195c3c Nov 03 19:01:20.210114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.210085 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.221724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.218201 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4"] Nov 03 19:01:20.246468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.246429 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.246584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.246568 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66h9v\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-kube-api-access-66h9v\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.365399 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.365326 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:20.541911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.541874 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" event={"ID":"51e8dbf6-c74b-4371-bc3d-0e0a6288522b","Type":"ContainerStarted","Data":"42e8fd9e8a10fcc294fb60c9450582f08935f406555ddc021ec233e604195c3c"} Nov 03 19:01:20.806980 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:20.806946 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b567e2b_d032_4733_a6bb_6531f0d149a2.slice/crio-9f6f23bfa09333b79104ac8111578b62fdd1afec7a39c5b150efa472964281ec WatchSource:0}: Error finding container 9f6f23bfa09333b79104ac8111578b62fdd1afec7a39c5b150efa472964281ec: Status 404 returned error can't find the container with id 9f6f23bfa09333b79104ac8111578b62fdd1afec7a39c5b150efa472964281ec Nov 03 19:01:20.808496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.808470 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6"] Nov 03 19:01:20.810007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.809959 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:20.810116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.810042 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:20.810116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:20.810081 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:21.249262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.249227 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb"] Nov 03 19:01:21.254180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.254156 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.297468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.297425 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ndl\" (UniqueName: \"kubernetes.io/projected/a7dd0e93-0807-42dd-a907-3c0796c68fdb-kube-api-access-z7ndl\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.297635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.297530 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.297745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.297723 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.310392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.310352 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb"] Nov 03 19:01:21.398927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.398889 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.399104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.399017 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ndl\" (UniqueName: \"kubernetes.io/projected/a7dd0e93-0807-42dd-a907-3c0796c68fdb-kube-api-access-z7ndl\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.399104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.399082 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.399871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.399842 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.401394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.401371 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.484611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.484577 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ndl\" (UniqueName: \"kubernetes.io/projected/a7dd0e93-0807-42dd-a907-3c0796c68fdb-kube-api-access-z7ndl\") pod \"maas-api-revoke-sub-8cde3da4-4ngfb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.577998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.577894 4948 generic.go:356] "Generic (PLEG): container finished" podID="51e8dbf6-c74b-4371-bc3d-0e0a6288522b" containerID="e00520243afdf354a366881af60bc943acda0070591b36c3b8a3786c4f1a1480" exitCode=0 Nov 03 19:01:21.577998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.577931 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" event={"ID":"51e8dbf6-c74b-4371-bc3d-0e0a6288522b","Type":"ContainerDied","Data":"e00520243afdf354a366881af60bc943acda0070591b36c3b8a3786c4f1a1480"} Nov 03 19:01:21.580593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.580565 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" event={"ID":"7b567e2b-d032-4733-a6bb-6531f0d149a2","Type":"ContainerStarted","Data":"5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765"} Nov 03 19:01:21.580740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.580621 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" event={"ID":"7b567e2b-d032-4733-a6bb-6531f0d149a2","Type":"ContainerStarted","Data":"9f6f23bfa09333b79104ac8111578b62fdd1afec7a39c5b150efa472964281ec"} Nov 03 19:01:21.627462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.626674 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:21.964727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:21.964014 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podStartSLOduration=2.963991822 podStartE2EDuration="2.963991822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:21.881612872 +0000 UTC m=+1411.364608787" watchObservedRunningTime="2026-11-03 19:01:21.963991822 +0000 UTC m=+1411.446987736" Nov 03 19:01:22.381550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:22.381090 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:22.431211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:22.428940 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 19:01:22.431211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:22.429003 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 19:01:22.932561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:22.932526 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb"] Nov 03 19:01:23.406744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.392594 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 19:01:23.406744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.392648 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 19:01:23.517474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.517078 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:23.642724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.631990 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-xks-ca\") pod \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " Nov 03 19:01:23.642724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.632060 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-kube-api-access-swfcv\" (UniqueName: \"kubernetes.io/projected/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-kube-api-access-swfcv\") pod \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " Nov 03 19:01:23.642724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.632099 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-openshift-ca\") pod \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\" (UID: \"51e8dbf6-c74b-4371-bc3d-0e0a6288522b\") " Nov 03 19:01:23.642724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.634430 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-openshift-ca" pod "51e8dbf6-c74b-4371-bc3d-0e0a6288522b" (UID: "51e8dbf6-c74b-4371-bc3d-0e0a6288522b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:23.666555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.666492 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-xks-ca" pod "51e8dbf6-c74b-4371-bc3d-0e0a6288522b" (UID: "51e8dbf6-c74b-4371-bc3d-0e0a6288522b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:23.674737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.673922 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-kube-api-access-swfcv" pod "51e8dbf6-c74b-4371-bc3d-0e0a6288522b" (UID: "51e8dbf6-c74b-4371-bc3d-0e0a6288522b"). InnerVolumeSpecName "kube-api-access-swfcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:23.883672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.883618 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:23.902766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.884281 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:23.902766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.884317 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfcv\" (UniqueName: \"kubernetes.io/projected/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-kube-api-access-swfcv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:23.902766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:23.884336 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e8dbf6-c74b-4371-bc3d-0e0a6288522b-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:24.429588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.426792 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 19:01:24.429588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.426862 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 19:01:24.433960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.433919 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:24.447436 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.447407 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" event={"ID":"a7dd0e93-0807-42dd-a907-3c0796c68fdb","Type":"ContainerStarted","Data":"c2a0feef82a6166d90ce0965ce37f441fbe3ba4a3d8c9ded2ab8065c5783363c"} Nov 03 19:01:24.447559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.447465 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" event={"ID":"a7dd0e93-0807-42dd-a907-3c0796c68fdb","Type":"ContainerStarted","Data":"e27efcc458a13459353261dc9b8f6b4ffecdb5b6eb2bc120bd75cbce6c0d3b30"} Nov 03 19:01:24.463745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.460333 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" Nov 03 19:01:24.463745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.460516 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4" event={"ID":"51e8dbf6-c74b-4371-bc3d-0e0a6288522b","Type":"ContainerDied","Data":"42e8fd9e8a10fcc294fb60c9450582f08935f406555ddc021ec233e604195c3c"} Nov 03 19:01:24.463745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.461771 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42e8fd9e8a10fcc294fb60c9450582f08935f406555ddc021ec233e604195c3c" Nov 03 19:01:24.490374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:24.490171 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:25.366128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.366088 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 19:01:25.366425 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.366387 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 19:01:25.502764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.502688 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68"] Nov 03 19:01:25.503155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.502837 4948 generic.go:356] "Generic (PLEG): container finished" podID="a7dd0e93-0807-42dd-a907-3c0796c68fdb" containerID="c2a0feef82a6166d90ce0965ce37f441fbe3ba4a3d8c9ded2ab8065c5783363c" exitCode=0 Nov 03 19:01:25.503961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.503439 4948 cpu_manager.go:414] "containerMap: removing container" podUID="51e8dbf6-c74b-4371-bc3d-0e0a6288522b" containerName="revoke-keys" Nov 03 19:01:25.503961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.503454 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e8dbf6-c74b-4371-bc3d-0e0a6288522b" containerName="revoke-keys" Nov 03 19:01:25.503961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.503594 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e8dbf6-c74b-4371-bc3d-0e0a6288522b" containerName="revoke-keys" Nov 03 19:01:25.508019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.507987 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" event={"ID":"a7dd0e93-0807-42dd-a907-3c0796c68fdb","Type":"ContainerDied","Data":"c2a0feef82a6166d90ce0965ce37f441fbe3ba4a3d8c9ded2ab8065c5783363c"} Nov 03 19:01:25.508129 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.508100 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.523154 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.523126 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68"] Nov 03 19:01:25.543714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.543666 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.543865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.543853 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.543936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.543914 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76v8\" (UniqueName: \"kubernetes.io/projected/1b67b136-e248-4823-978a-5d355a8ef553-kube-api-access-p76v8\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.645024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.644959 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.645161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.645031 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76v8\" (UniqueName: \"kubernetes.io/projected/1b67b136-e248-4823-978a-5d355a8ef553-kube-api-access-p76v8\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.645161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.645070 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.645661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.645633 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.647479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.647458 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.722033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.721900 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76v8\" (UniqueName: \"kubernetes.io/projected/1b67b136-e248-4823-978a-5d355a8ef553-kube-api-access-p76v8\") pod \"maas-api-revoke-sub-65a78933-9ng68\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.827945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.827921 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:25.840210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.838141 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:25.850214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.850186 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-openshift-ca\") pod \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " Nov 03 19:01:25.850320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.850297 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dd0e93-0807-42dd-a907-3c0796c68fdb-kube-api-access-z7ndl\" (UniqueName: \"kubernetes.io/projected/a7dd0e93-0807-42dd-a907-3c0796c68fdb-kube-api-access-z7ndl\") pod \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " Nov 03 19:01:25.850395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.850334 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-xks-ca\") pod \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\" (UID: \"a7dd0e93-0807-42dd-a907-3c0796c68fdb\") " Nov 03 19:01:25.850537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.850512 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-openshift-ca" pod "a7dd0e93-0807-42dd-a907-3c0796c68fdb" (UID: "a7dd0e93-0807-42dd-a907-3c0796c68fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:25.862727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.850921 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:25.904275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.904197 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-xks-ca" pod "a7dd0e93-0807-42dd-a907-3c0796c68fdb" (UID: "a7dd0e93-0807-42dd-a907-3c0796c68fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:25.915347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:25.915294 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dd0e93-0807-42dd-a907-3c0796c68fdb-kube-api-access-z7ndl" pod "a7dd0e93-0807-42dd-a907-3c0796c68fdb" (UID: "a7dd0e93-0807-42dd-a907-3c0796c68fdb"). InnerVolumeSpecName "kube-api-access-z7ndl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:26.003509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.003449 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ndl\" (UniqueName: \"kubernetes.io/projected/a7dd0e93-0807-42dd-a907-3c0796c68fdb-kube-api-access-z7ndl\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:26.003774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.003753 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dd0e93-0807-42dd-a907-3c0796c68fdb-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:26.299776 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:26.299747 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b67b136_e248_4823_978a_5d355a8ef553.slice/crio-59f4ba20dc3f94c446ce6847a98da061d71f2e472a34222cdcf611031c002300 WatchSource:0}: Error finding container 59f4ba20dc3f94c446ce6847a98da061d71f2e472a34222cdcf611031c002300: Status 404 returned error can't find the container with id 59f4ba20dc3f94c446ce6847a98da061d71f2e472a34222cdcf611031c002300 Nov 03 19:01:26.302608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.302578 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68"] Nov 03 19:01:26.333043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.333015 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd"] Nov 03 19:01:26.333666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.333651 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a7dd0e93-0807-42dd-a907-3c0796c68fdb" containerName="revoke-keys" Nov 03 19:01:26.333666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.333666 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dd0e93-0807-42dd-a907-3c0796c68fdb" containerName="revoke-keys" Nov 03 19:01:26.333824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.333814 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dd0e93-0807-42dd-a907-3c0796c68fdb" containerName="revoke-keys" Nov 03 19:01:26.338769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.338750 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.386019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.383812 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 19:01:26.386019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.383870 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 19:01:26.397883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.397840 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd"] Nov 03 19:01:26.409304 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.409265 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e5e64-027b-4665-8508-560326c5a39b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.409459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.409389 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e5e64-027b-4665-8508-560326c5a39b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.409516 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.409468 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2vj\" (UniqueName: \"kubernetes.io/projected/514e5e64-027b-4665-8508-560326c5a39b-kube-api-access-hk2vj\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.510480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.510440 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e5e64-027b-4665-8508-560326c5a39b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.510939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.510543 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2vj\" (UniqueName: \"kubernetes.io/projected/514e5e64-027b-4665-8508-560326c5a39b-kube-api-access-hk2vj\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.510939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.510744 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e5e64-027b-4665-8508-560326c5a39b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.511552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.511529 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e5e64-027b-4665-8508-560326c5a39b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.512853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.512835 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e5e64-027b-4665-8508-560326c5a39b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.514846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.514825 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" Nov 03 19:01:26.514954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.514845 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb" event={"ID":"a7dd0e93-0807-42dd-a907-3c0796c68fdb","Type":"ContainerDied","Data":"e27efcc458a13459353261dc9b8f6b4ffecdb5b6eb2bc120bd75cbce6c0d3b30"} Nov 03 19:01:26.514954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.514899 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27efcc458a13459353261dc9b8f6b4ffecdb5b6eb2bc120bd75cbce6c0d3b30" Nov 03 19:01:26.516261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.516237 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" event={"ID":"1b67b136-e248-4823-978a-5d355a8ef553","Type":"ContainerStarted","Data":"59f4ba20dc3f94c446ce6847a98da061d71f2e472a34222cdcf611031c002300"} Nov 03 19:01:26.589910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.588419 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2vj\" (UniqueName: \"kubernetes.io/projected/514e5e64-027b-4665-8508-560326c5a39b-kube-api-access-hk2vj\") pod \"maas-api-revoke-sub-b1d591c0-sh6rd\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:26.699872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:26.699838 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:27.148934 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:27.148896 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514e5e64_027b_4665_8508_560326c5a39b.slice/crio-ec55e84493b37c41edac39ec8717c1eb83bdee0b268addc2d53fdf0bed554df9 WatchSource:0}: Error finding container ec55e84493b37c41edac39ec8717c1eb83bdee0b268addc2d53fdf0bed554df9: Status 404 returned error can't find the container with id ec55e84493b37c41edac39ec8717c1eb83bdee0b268addc2d53fdf0bed554df9 Nov 03 19:01:27.159803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:27.159759 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd"] Nov 03 19:01:27.369448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:27.368992 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 19:01:27.369448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:27.369052 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 19:01:27.527159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:27.526897 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" event={"ID":"514e5e64-027b-4665-8508-560326c5a39b","Type":"ContainerStarted","Data":"ec55e84493b37c41edac39ec8717c1eb83bdee0b268addc2d53fdf0bed554df9"} Nov 03 19:01:27.531935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:27.531909 4948 generic.go:356] "Generic (PLEG): container finished" podID="1b67b136-e248-4823-978a-5d355a8ef553" containerID="f290352c51a53a7b699492748fc5ea9b8624ccc3c41944eb20b972825e411403" exitCode=0 Nov 03 19:01:27.532097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:27.532081 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" event={"ID":"1b67b136-e248-4823-978a-5d355a8ef553","Type":"ContainerDied","Data":"f290352c51a53a7b699492748fc5ea9b8624ccc3c41944eb20b972825e411403"} Nov 03 19:01:27.534684 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:01:27.534658 4948 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b67b136_e248_4823_978a_5d355a8ef553.slice/crio-f290352c51a53a7b699492748fc5ea9b8624ccc3c41944eb20b972825e411403.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b67b136_e248_4823_978a_5d355a8ef553.slice/crio-conmon-f290352c51a53a7b699492748fc5ea9b8624ccc3c41944eb20b972825e411403.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:01:28.366215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.366119 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 19:01:28.366215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.366200 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 19:01:28.544392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.544352 4948 generic.go:356] "Generic (PLEG): container finished" podID="514e5e64-027b-4665-8508-560326c5a39b" containerID="7f4fe7d2291bffc5f8dca74e01b91b1363bda7f774a53ecc12f369e147c79109" exitCode=0 Nov 03 19:01:28.544816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.544433 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" event={"ID":"514e5e64-027b-4665-8508-560326c5a39b","Type":"ContainerDied","Data":"7f4fe7d2291bffc5f8dca74e01b91b1363bda7f774a53ecc12f369e147c79109"} Nov 03 19:01:28.744505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.744477 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:28.851694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.851657 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-xks-ca\") pod \"1b67b136-e248-4823-978a-5d355a8ef553\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " Nov 03 19:01:28.851961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.851939 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-openshift-ca\") pod \"1b67b136-e248-4823-978a-5d355a8ef553\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " Nov 03 19:01:28.852040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.852015 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b67b136-e248-4823-978a-5d355a8ef553-kube-api-access-p76v8\" (UniqueName: \"kubernetes.io/projected/1b67b136-e248-4823-978a-5d355a8ef553-kube-api-access-p76v8\") pod \"1b67b136-e248-4823-978a-5d355a8ef553\" (UID: \"1b67b136-e248-4823-978a-5d355a8ef553\") " Nov 03 19:01:28.852402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.852360 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-openshift-ca" pod "1b67b136-e248-4823-978a-5d355a8ef553" (UID: "1b67b136-e248-4823-978a-5d355a8ef553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:28.853125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.853095 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:28.855785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.855756 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b67b136-e248-4823-978a-5d355a8ef553-kube-api-access-p76v8" pod "1b67b136-e248-4823-978a-5d355a8ef553" (UID: "1b67b136-e248-4823-978a-5d355a8ef553"). InnerVolumeSpecName "kube-api-access-p76v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:28.855893 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.855879 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-xks-ca" pod "1b67b136-e248-4823-978a-5d355a8ef553" (UID: "1b67b136-e248-4823-978a-5d355a8ef553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:28.954880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.954846 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76v8\" (UniqueName: \"kubernetes.io/projected/1b67b136-e248-4823-978a-5d355a8ef553-kube-api-access-p76v8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:28.954880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:28.954876 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b67b136-e248-4823-978a-5d355a8ef553-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:29.371550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.371468 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:29.371977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.371948 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:29.372802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.372784 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:01:29.555648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.555619 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" Nov 03 19:01:29.556160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.555649 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68" event={"ID":"1b67b136-e248-4823-978a-5d355a8ef553","Type":"ContainerDied","Data":"59f4ba20dc3f94c446ce6847a98da061d71f2e472a34222cdcf611031c002300"} Nov 03 19:01:29.556160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.555735 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f4ba20dc3f94c446ce6847a98da061d71f2e472a34222cdcf611031c002300" Nov 03 19:01:29.750861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.750817 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:29.919358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.916789 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514e5e64-027b-4665-8508-560326c5a39b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e5e64-027b-4665-8508-560326c5a39b-maas-api-openshift-ca\") pod \"514e5e64-027b-4665-8508-560326c5a39b\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " Nov 03 19:01:29.919358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.916872 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514e5e64-027b-4665-8508-560326c5a39b-kube-api-access-hk2vj\" (UniqueName: \"kubernetes.io/projected/514e5e64-027b-4665-8508-560326c5a39b-kube-api-access-hk2vj\") pod \"514e5e64-027b-4665-8508-560326c5a39b\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " Nov 03 19:01:29.919358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.917088 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514e5e64-027b-4665-8508-560326c5a39b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e5e64-027b-4665-8508-560326c5a39b-maas-api-xks-ca\") pod \"514e5e64-027b-4665-8508-560326c5a39b\" (UID: \"514e5e64-027b-4665-8508-560326c5a39b\") " Nov 03 19:01:29.921897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.921368 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514e5e64-027b-4665-8508-560326c5a39b-maas-api-openshift-ca" pod "514e5e64-027b-4665-8508-560326c5a39b" (UID: "514e5e64-027b-4665-8508-560326c5a39b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:29.921897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.921506 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514e5e64-027b-4665-8508-560326c5a39b-maas-api-xks-ca" pod "514e5e64-027b-4665-8508-560326c5a39b" (UID: "514e5e64-027b-4665-8508-560326c5a39b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:29.921897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:29.921638 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514e5e64-027b-4665-8508-560326c5a39b-kube-api-access-hk2vj" pod "514e5e64-027b-4665-8508-560326c5a39b" (UID: "514e5e64-027b-4665-8508-560326c5a39b"). InnerVolumeSpecName "kube-api-access-hk2vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:30.019585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:30.019509 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514e5e64-027b-4665-8508-560326c5a39b-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:30.019585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:30.019539 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2vj\" (UniqueName: \"kubernetes.io/projected/514e5e64-027b-4665-8508-560326c5a39b-kube-api-access-hk2vj\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:30.019585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:30.019552 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514e5e64-027b-4665-8508-560326c5a39b-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:30.567405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:30.567374 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" Nov 03 19:01:30.567951 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:30.567408 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd" event={"ID":"514e5e64-027b-4665-8508-560326c5a39b","Type":"ContainerDied","Data":"ec55e84493b37c41edac39ec8717c1eb83bdee0b268addc2d53fdf0bed554df9"} Nov 03 19:01:30.567951 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:30.567497 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec55e84493b37c41edac39ec8717c1eb83bdee0b268addc2d53fdf0bed554df9" Nov 03 19:01:31.053046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:31.053012 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g"] Nov 03 19:01:31.057935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:31.057907 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f72f05d-tx59g"] Nov 03 19:01:32.056104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.056071 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr"] Nov 03 19:01:32.057057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.057035 4948 cpu_manager.go:414] "containerMap: removing container" podUID="514e5e64-027b-4665-8508-560326c5a39b" containerName="revoke-keys" Nov 03 19:01:32.057106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.057060 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514e5e64-027b-4665-8508-560326c5a39b" containerName="revoke-keys" Nov 03 19:01:32.057142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.057130 4948 cpu_manager.go:414] "containerMap: removing container" podUID="1b67b136-e248-4823-978a-5d355a8ef553" containerName="revoke-keys" Nov 03 19:01:32.057142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.057138 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b67b136-e248-4823-978a-5d355a8ef553" containerName="revoke-keys" Nov 03 19:01:32.057287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.057276 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="514e5e64-027b-4665-8508-560326c5a39b" containerName="revoke-keys" Nov 03 19:01:32.057331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.057300 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b67b136-e248-4823-978a-5d355a8ef553" containerName="revoke-keys" Nov 03 19:01:32.061900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.061879 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.113940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.113906 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr"] Nov 03 19:01:32.142415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.142383 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfsd\" (UniqueName: \"kubernetes.io/projected/3eb29dd1-c13a-4805-9b60-521f3bf2a842-kube-api-access-9pfsd\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.142589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.142573 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.142688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.142671 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.158794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.158768 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:01:32.158967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.158804 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:01:32.158967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.158804 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:01:32.158967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.158848 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:01:32.243432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.243395 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfsd\" (UniqueName: \"kubernetes.io/projected/3eb29dd1-c13a-4805-9b60-521f3bf2a842-kube-api-access-9pfsd\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.243621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.243558 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.243692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.243645 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.244433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.244407 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.246107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.246086 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.305487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.305455 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfsd\" (UniqueName: \"kubernetes.io/projected/3eb29dd1-c13a-4805-9b60-521f3bf2a842-kube-api-access-9pfsd\") pod \"maas-api-revoke-sub-be375354-n9chr\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.376803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.376717 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:32.378527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.378496 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm"] Nov 03 19:01:32.382930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.382910 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.441403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.441364 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm"] Nov 03 19:01:32.446041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.445864 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7vl\" (UniqueName: \"kubernetes.io/projected/49ff101f-eb04-48c3-9309-61bbd285a8fa-kube-api-access-gw7vl\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.446160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.446071 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.446206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.446188 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.547835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.547802 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7vl\" (UniqueName: \"kubernetes.io/projected/49ff101f-eb04-48c3-9309-61bbd285a8fa-kube-api-access-gw7vl\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.548008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.547870 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.548008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.547936 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.548666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.548639 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.550517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.550497 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.609995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.609963 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7vl\" (UniqueName: \"kubernetes.io/projected/49ff101f-eb04-48c3-9309-61bbd285a8fa-kube-api-access-gw7vl\") pod \"maas-api-revoke-sub-1c6dc9bc-h86cm\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.722964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.722935 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:32.767777 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:32.767746 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb29dd1_c13a_4805_9b60_521f3bf2a842.slice/crio-204fd82220f558556af2bd96c70b44c8b356804139009f89916b14c58229ac01 WatchSource:0}: Error finding container 204fd82220f558556af2bd96c70b44c8b356804139009f89916b14c58229ac01: Status 404 returned error can't find the container with id 204fd82220f558556af2bd96c70b44c8b356804139009f89916b14c58229ac01 Nov 03 19:01:32.775996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.775964 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f59cf4-27b4-4c6d-93bc-62682bf0a3f6" path="/var/lib/kubelet/pods/81f59cf4-27b4-4c6d-93bc-62682bf0a3f6/volumes" Nov 03 19:01:32.776471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:32.776449 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr"] Nov 03 19:01:32.802220 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:01:32.802186 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:01:33.126057 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:33.126026 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ff101f_eb04_48c3_9309_61bbd285a8fa.slice/crio-4ef409b9228eb9fca0957b1f8f5d1fbe65930aeac3c859c34de2cf3aced0b488 WatchSource:0}: Error finding container 4ef409b9228eb9fca0957b1f8f5d1fbe65930aeac3c859c34de2cf3aced0b488: Status 404 returned error can't find the container with id 4ef409b9228eb9fca0957b1f8f5d1fbe65930aeac3c859c34de2cf3aced0b488 Nov 03 19:01:33.128426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:33.128404 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm"] Nov 03 19:01:33.599911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:33.599865 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" event={"ID":"3eb29dd1-c13a-4805-9b60-521f3bf2a842","Type":"ContainerStarted","Data":"cbfce7620238b95876ef6e6dab9c9ec1ccef426c60bbd976077f489a28247ae5"} Nov 03 19:01:33.600080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:33.599935 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" event={"ID":"3eb29dd1-c13a-4805-9b60-521f3bf2a842","Type":"ContainerStarted","Data":"204fd82220f558556af2bd96c70b44c8b356804139009f89916b14c58229ac01"} Nov 03 19:01:33.601572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:33.601548 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" event={"ID":"49ff101f-eb04-48c3-9309-61bbd285a8fa","Type":"ContainerStarted","Data":"cadaf38fb50408f706b8719b9b0054fb2f8beafd5e02bd49e3bceca2f090bb46"} Nov 03 19:01:33.601676 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:33.601588 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" event={"ID":"49ff101f-eb04-48c3-9309-61bbd285a8fa","Type":"ContainerStarted","Data":"4ef409b9228eb9fca0957b1f8f5d1fbe65930aeac3c859c34de2cf3aced0b488"} Nov 03 19:01:33.720155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:33.720106 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" podStartSLOduration=1.720092394 podStartE2EDuration="1.720092394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:33.717493441 +0000 UTC m=+1423.200489356" watchObservedRunningTime="2026-11-03 19:01:33.720092394 +0000 UTC m=+1423.203088308" Nov 03 19:01:33.838661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:33.838606 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" podStartSLOduration=1.838592007 podStartE2EDuration="1.838592007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:33.835398216 +0000 UTC m=+1423.318394140" watchObservedRunningTime="2026-11-03 19:01:33.838592007 +0000 UTC m=+1423.321587952" Nov 03 19:01:34.367575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.367501 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv"] Nov 03 19:01:34.372444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.372418 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.427330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.427294 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv"] Nov 03 19:01:34.469271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.469239 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.469445 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.469431 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.469554 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.469534 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5glg6\" (UniqueName: \"kubernetes.io/projected/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-kube-api-access-5glg6\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.570463 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.570414 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.570728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.570521 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5glg6\" (UniqueName: \"kubernetes.io/projected/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-kube-api-access-5glg6\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.570728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.570624 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.571334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.571304 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.573280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.573254 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.613365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.613327 4948 generic.go:356] "Generic (PLEG): container finished" podID="3eb29dd1-c13a-4805-9b60-521f3bf2a842" containerID="cbfce7620238b95876ef6e6dab9c9ec1ccef426c60bbd976077f489a28247ae5" exitCode=0 Nov 03 19:01:34.613550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.613408 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" event={"ID":"3eb29dd1-c13a-4805-9b60-521f3bf2a842","Type":"ContainerDied","Data":"cbfce7620238b95876ef6e6dab9c9ec1ccef426c60bbd976077f489a28247ae5"} Nov 03 19:01:34.615207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.615182 4948 generic.go:356] "Generic (PLEG): container finished" podID="49ff101f-eb04-48c3-9309-61bbd285a8fa" containerID="cadaf38fb50408f706b8719b9b0054fb2f8beafd5e02bd49e3bceca2f090bb46" exitCode=0 Nov 03 19:01:34.615337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.615227 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" event={"ID":"49ff101f-eb04-48c3-9309-61bbd285a8fa","Type":"ContainerDied","Data":"cadaf38fb50408f706b8719b9b0054fb2f8beafd5e02bd49e3bceca2f090bb46"} Nov 03 19:01:34.633365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.633298 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5glg6\" (UniqueName: \"kubernetes.io/projected/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-kube-api-access-5glg6\") pod \"maas-api-revoke-sub-3aaa9e67-knpbv\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:34.687060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:34.687032 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:35.072048 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:35.072012 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ff9ac2_580c_4828_9d05_a2b48c2b57d9.slice/crio-0c7c1acb5e130a8a9795e6004c20c1ea1f4b9cac4e8a410d758f873eaf5ef504 WatchSource:0}: Error finding container 0c7c1acb5e130a8a9795e6004c20c1ea1f4b9cac4e8a410d758f873eaf5ef504: Status 404 returned error can't find the container with id 0c7c1acb5e130a8a9795e6004c20c1ea1f4b9cac4e8a410d758f873eaf5ef504 Nov 03 19:01:35.073317 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.073292 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv"] Nov 03 19:01:35.160930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.160891 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs"] Nov 03 19:01:35.167079 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.167061 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179077 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179050 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179098 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179186 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgc4f\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-kube-api-access-zgc4f\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179211 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179251 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179362 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179477 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179572 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1e7095-d0f7-4548-ae01-1ea485ece334-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.179659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.179642 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.237228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.236860 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs"] Nov 03 19:01:35.281266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281227 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281308 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1e7095-d0f7-4548-ae01-1ea485ece334-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281461 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281659 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281689 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281711 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281737 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgc4f\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-kube-api-access-zgc4f\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281769 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281826 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.281994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.281858 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.282278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.282179 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.282656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.282627 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.282780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.282691 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.283159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.283134 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1e7095-d0f7-4548-ae01-1ea485ece334-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.284560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.284535 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.284774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.284752 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.306190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.306156 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj"] Nov 03 19:01:35.312967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.312947 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.347631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.347555 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.347787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.347756 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgc4f\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-kube-api-access-zgc4f\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.360383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.360339 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj"] Nov 03 19:01:35.383553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.383521 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.383992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.383659 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwxb\" (UniqueName: \"kubernetes.io/projected/874390a6-68da-4a9c-9680-64cc1ebc2157-kube-api-access-4hwxb\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.383992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.383804 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.481227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.481196 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:35.484522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.484499 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.484615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.484600 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwxb\" (UniqueName: \"kubernetes.io/projected/874390a6-68da-4a9c-9680-64cc1ebc2157-kube-api-access-4hwxb\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.484802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.484782 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.485223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.485205 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.487343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.487319 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.548080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.548050 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwxb\" (UniqueName: \"kubernetes.io/projected/874390a6-68da-4a9c-9680-64cc1ebc2157-kube-api-access-4hwxb\") pod \"maas-api-revoke-sub-dd8e329f-x4kbj\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.627359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.627276 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:35.631170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.631125 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" event={"ID":"57ff9ac2-580c-4828-9d05-a2b48c2b57d9","Type":"ContainerStarted","Data":"af189019f9c484918d4d7b1c06b74fe961671f0271b6cf818b7b74335dfe7ccb"} Nov 03 19:01:35.631329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.631204 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" event={"ID":"57ff9ac2-580c-4828-9d05-a2b48c2b57d9","Type":"ContainerStarted","Data":"0c7c1acb5e130a8a9795e6004c20c1ea1f4b9cac4e8a410d758f873eaf5ef504"} Nov 03 19:01:35.753826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.753781 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" podStartSLOduration=1.753745437 podStartE2EDuration="1.753745437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:35.75102535 +0000 UTC m=+1425.234021277" watchObservedRunningTime="2026-11-03 19:01:35.753745437 +0000 UTC m=+1425.236741350" Nov 03 19:01:35.831954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.831926 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:35.877510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.877439 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:35.888798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.888764 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-openshift-ca\") pod \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " Nov 03 19:01:35.888965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.888899 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb29dd1-c13a-4805-9b60-521f3bf2a842-kube-api-access-9pfsd\" (UniqueName: \"kubernetes.io/projected/3eb29dd1-c13a-4805-9b60-521f3bf2a842-kube-api-access-9pfsd\") pod \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " Nov 03 19:01:35.889043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.888985 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-xks-ca\") pod \"49ff101f-eb04-48c3-9309-61bbd285a8fa\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " Nov 03 19:01:35.889166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.889148 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-xks-ca\") pod \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\" (UID: \"3eb29dd1-c13a-4805-9b60-521f3bf2a842\") " Nov 03 19:01:35.889243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.889204 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ff101f-eb04-48c3-9309-61bbd285a8fa-kube-api-access-gw7vl\" (UniqueName: \"kubernetes.io/projected/49ff101f-eb04-48c3-9309-61bbd285a8fa-kube-api-access-gw7vl\") pod \"49ff101f-eb04-48c3-9309-61bbd285a8fa\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " Nov 03 19:01:35.889306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.889239 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-openshift-ca\") pod \"49ff101f-eb04-48c3-9309-61bbd285a8fa\" (UID: \"49ff101f-eb04-48c3-9309-61bbd285a8fa\") " Nov 03 19:01:35.889467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.889145 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-openshift-ca" pod "3eb29dd1-c13a-4805-9b60-521f3bf2a842" (UID: "3eb29dd1-c13a-4805-9b60-521f3bf2a842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:35.889864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.889839 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:35.890207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.890180 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-openshift-ca" pod "49ff101f-eb04-48c3-9309-61bbd285a8fa" (UID: "49ff101f-eb04-48c3-9309-61bbd285a8fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:35.893336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.893308 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb29dd1-c13a-4805-9b60-521f3bf2a842-kube-api-access-9pfsd" pod "3eb29dd1-c13a-4805-9b60-521f3bf2a842" (UID: "3eb29dd1-c13a-4805-9b60-521f3bf2a842"). InnerVolumeSpecName "kube-api-access-9pfsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:35.893336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.893325 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-xks-ca" pod "49ff101f-eb04-48c3-9309-61bbd285a8fa" (UID: "49ff101f-eb04-48c3-9309-61bbd285a8fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:35.893458 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.893426 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ff101f-eb04-48c3-9309-61bbd285a8fa-kube-api-access-gw7vl" pod "49ff101f-eb04-48c3-9309-61bbd285a8fa" (UID: "49ff101f-eb04-48c3-9309-61bbd285a8fa"). InnerVolumeSpecName "kube-api-access-gw7vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:35.893943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.893917 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-xks-ca" pod "3eb29dd1-c13a-4805-9b60-521f3bf2a842" (UID: "3eb29dd1-c13a-4805-9b60-521f3bf2a842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:35.921196 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:35.921159 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1e7095_d0f7_4548_ae01_1ea485ece334.slice/crio-65e06d8340bc1502b212ea07df101d0767ed9fa4e6c25289ba18dcf5ea7b2b57 WatchSource:0}: Error finding container 65e06d8340bc1502b212ea07df101d0767ed9fa4e6c25289ba18dcf5ea7b2b57: Status 404 returned error can't find the container with id 65e06d8340bc1502b212ea07df101d0767ed9fa4e6c25289ba18dcf5ea7b2b57 Nov 03 19:01:35.922661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.922632 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs"] Nov 03 19:01:35.923949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.923921 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:35.924009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.923993 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:35.924053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.924040 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:35.991535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.991503 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:35.991692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.991539 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb29dd1-c13a-4805-9b60-521f3bf2a842-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:35.991692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.991555 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7vl\" (UniqueName: \"kubernetes.io/projected/49ff101f-eb04-48c3-9309-61bbd285a8fa-kube-api-access-gw7vl\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:35.991692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.991570 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ff101f-eb04-48c3-9309-61bbd285a8fa-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:35.991692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:35.991586 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfsd\" (UniqueName: \"kubernetes.io/projected/3eb29dd1-c13a-4805-9b60-521f3bf2a842-kube-api-access-9pfsd\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:36.049017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.048712 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj"] Nov 03 19:01:36.641790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.641751 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" event={"ID":"874390a6-68da-4a9c-9680-64cc1ebc2157","Type":"ContainerStarted","Data":"7cfaf4e9803e8766f7657da78893a35a5c2eae3ee4ee4dd1a838bfdaee1f70e0"} Nov 03 19:01:36.642169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.641829 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" event={"ID":"874390a6-68da-4a9c-9680-64cc1ebc2157","Type":"ContainerStarted","Data":"3c280813ee52d22de188c23f29ef945003b4b3dd60fd1ab7d54bc58e6779acb0"} Nov 03 19:01:36.643467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.643447 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" Nov 03 19:01:36.643572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.643501 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm" event={"ID":"49ff101f-eb04-48c3-9309-61bbd285a8fa","Type":"ContainerDied","Data":"4ef409b9228eb9fca0957b1f8f5d1fbe65930aeac3c859c34de2cf3aced0b488"} Nov 03 19:01:36.643572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.643550 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef409b9228eb9fca0957b1f8f5d1fbe65930aeac3c859c34de2cf3aced0b488" Nov 03 19:01:36.645364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.645343 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" event={"ID":"9b1e7095-d0f7-4548-ae01-1ea485ece334","Type":"ContainerStarted","Data":"def6eb8ca2e56203760b60e71e6ed5904487bb785fd0025445645e15f820a207"} Nov 03 19:01:36.645444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.645385 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" event={"ID":"9b1e7095-d0f7-4548-ae01-1ea485ece334","Type":"ContainerStarted","Data":"65e06d8340bc1502b212ea07df101d0767ed9fa4e6c25289ba18dcf5ea7b2b57"} Nov 03 19:01:36.647286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.647265 4948 generic.go:356] "Generic (PLEG): container finished" podID="57ff9ac2-580c-4828-9d05-a2b48c2b57d9" containerID="af189019f9c484918d4d7b1c06b74fe961671f0271b6cf818b7b74335dfe7ccb" exitCode=0 Nov 03 19:01:36.647402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.647351 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" event={"ID":"57ff9ac2-580c-4828-9d05-a2b48c2b57d9","Type":"ContainerDied","Data":"af189019f9c484918d4d7b1c06b74fe961671f0271b6cf818b7b74335dfe7ccb"} Nov 03 19:01:36.648868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.648851 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" event={"ID":"3eb29dd1-c13a-4805-9b60-521f3bf2a842","Type":"ContainerDied","Data":"204fd82220f558556af2bd96c70b44c8b356804139009f89916b14c58229ac01"} Nov 03 19:01:36.648942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.648875 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr" Nov 03 19:01:36.648981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.648881 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="204fd82220f558556af2bd96c70b44c8b356804139009f89916b14c58229ac01" Nov 03 19:01:36.769104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:36.768987 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" podStartSLOduration=1.768967751 podStartE2EDuration="1.768967751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:36.759432727 +0000 UTC m=+1426.242428643" watchObservedRunningTime="2026-11-03 19:01:36.768967751 +0000 UTC m=+1426.251963667" Nov 03 19:01:37.081551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:37.081428 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podStartSLOduration=2.081404603 podStartE2EDuration="2.081404603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:37.067542122 +0000 UTC m=+1426.550538039" watchObservedRunningTime="2026-11-03 19:01:37.081404603 +0000 UTC m=+1426.564400518" Nov 03 19:01:37.482194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:37.482147 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:37.483875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:37.483846 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:01:37.484017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:37.483905 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 19:01:37.674723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:37.670991 4948 generic.go:356] "Generic (PLEG): container finished" podID="874390a6-68da-4a9c-9680-64cc1ebc2157" containerID="7cfaf4e9803e8766f7657da78893a35a5c2eae3ee4ee4dd1a838bfdaee1f70e0" exitCode=0 Nov 03 19:01:37.674723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:37.671684 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" event={"ID":"874390a6-68da-4a9c-9680-64cc1ebc2157","Type":"ContainerDied","Data":"7cfaf4e9803e8766f7657da78893a35a5c2eae3ee4ee4dd1a838bfdaee1f70e0"} Nov 03 19:01:37.916834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:37.916806 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:38.015649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.015567 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-openshift-ca\") pod \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " Nov 03 19:01:38.015845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.015823 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-kube-api-access-5glg6\" (UniqueName: \"kubernetes.io/projected/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-kube-api-access-5glg6\") pod \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " Nov 03 19:01:38.015912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.015868 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-xks-ca\") pod \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\" (UID: \"57ff9ac2-580c-4828-9d05-a2b48c2b57d9\") " Nov 03 19:01:38.015967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.015944 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-openshift-ca" pod "57ff9ac2-580c-4828-9d05-a2b48c2b57d9" (UID: "57ff9ac2-580c-4828-9d05-a2b48c2b57d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:38.016376 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.016354 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:38.018025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.017994 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-xks-ca" pod "57ff9ac2-580c-4828-9d05-a2b48c2b57d9" (UID: "57ff9ac2-580c-4828-9d05-a2b48c2b57d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:38.018133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.018107 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-kube-api-access-5glg6" pod "57ff9ac2-580c-4828-9d05-a2b48c2b57d9" (UID: "57ff9ac2-580c-4828-9d05-a2b48c2b57d9"). InnerVolumeSpecName "kube-api-access-5glg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:38.092815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.092774 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk"] Nov 03 19:01:38.098346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.098312 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2709ff2b-dmxhk"] Nov 03 19:01:38.106688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.106633 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v"] Nov 03 19:01:38.112361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.112340 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c738e92-rmn9v"] Nov 03 19:01:38.118839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.118810 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5glg6\" (UniqueName: \"kubernetes.io/projected/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-kube-api-access-5glg6\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:38.118945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.118845 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff9ac2-580c-4828-9d05-a2b48c2b57d9-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.138542 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c"] Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139473 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3eb29dd1-c13a-4805-9b60-521f3bf2a842" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139492 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb29dd1-c13a-4805-9b60-521f3bf2a842" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139534 4948 cpu_manager.go:414] "containerMap: removing container" podUID="49ff101f-eb04-48c3-9309-61bbd285a8fa" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139543 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ff101f-eb04-48c3-9309-61bbd285a8fa" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139565 4948 cpu_manager.go:414] "containerMap: removing container" podUID="57ff9ac2-580c-4828-9d05-a2b48c2b57d9" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139572 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ff9ac2-580c-4828-9d05-a2b48c2b57d9" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139780 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ff101f-eb04-48c3-9309-61bbd285a8fa" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139799 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ff9ac2-580c-4828-9d05-a2b48c2b57d9" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.139816 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb29dd1-c13a-4805-9b60-521f3bf2a842" containerName="revoke-keys" Nov 03 19:01:38.157422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.144516 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.203617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.203588 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c"] Nov 03 19:01:38.220114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.220081 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.220250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.220166 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.220408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.220389 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bknj\" (UniqueName: \"kubernetes.io/projected/e056629f-76b0-49a7-b632-48a5729bf91f-kube-api-access-8bknj\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.321471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.321391 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.321471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.321437 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.321671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.321532 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bknj\" (UniqueName: \"kubernetes.io/projected/e056629f-76b0-49a7-b632-48a5729bf91f-kube-api-access-8bknj\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.322234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.322205 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.324012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.323980 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.387602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.387569 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bknj\" (UniqueName: \"kubernetes.io/projected/e056629f-76b0-49a7-b632-48a5729bf91f-kube-api-access-8bknj\") pod \"maas-api-revoke-sub-1688ae35-z227c\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.483722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.483062 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:01:38.483722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.483134 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 19:01:38.493946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.493909 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:38.694668 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.694634 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" Nov 03 19:01:38.695330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.695300 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv" event={"ID":"57ff9ac2-580c-4828-9d05-a2b48c2b57d9","Type":"ContainerDied","Data":"0c7c1acb5e130a8a9795e6004c20c1ea1f4b9cac4e8a410d758f873eaf5ef504"} Nov 03 19:01:38.695435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.695378 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7c1acb5e130a8a9795e6004c20c1ea1f4b9cac4e8a410d758f873eaf5ef504" Nov 03 19:01:38.786023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.785984 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04889421-f5e1-4a7f-b19d-2da6488c4b8e" path="/var/lib/kubelet/pods/04889421-f5e1-4a7f-b19d-2da6488c4b8e/volumes" Nov 03 19:01:38.786786 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.786759 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df5c7ca-9b8f-43e8-b283-62238b64cd6b" path="/var/lib/kubelet/pods/3df5c7ca-9b8f-43e8-b283-62238b64cd6b/volumes" Nov 03 19:01:38.908832 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:38.908803 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode056629f_76b0_49a7_b632_48a5729bf91f.slice/crio-b0649d5a59f1b011fc345a06281637fb4ba0717ff2995717cd8ca1288ad324a9 WatchSource:0}: Error finding container b0649d5a59f1b011fc345a06281637fb4ba0717ff2995717cd8ca1288ad324a9: Status 404 returned error can't find the container with id b0649d5a59f1b011fc345a06281637fb4ba0717ff2995717cd8ca1288ad324a9 Nov 03 19:01:38.910682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.910592 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c"] Nov 03 19:01:38.938213 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:38.938190 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:39.032045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.032005 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-xks-ca\") pod \"874390a6-68da-4a9c-9680-64cc1ebc2157\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " Nov 03 19:01:39.032243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.032095 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874390a6-68da-4a9c-9680-64cc1ebc2157-kube-api-access-4hwxb\" (UniqueName: \"kubernetes.io/projected/874390a6-68da-4a9c-9680-64cc1ebc2157-kube-api-access-4hwxb\") pod \"874390a6-68da-4a9c-9680-64cc1ebc2157\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " Nov 03 19:01:39.032243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.032227 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-openshift-ca\") pod \"874390a6-68da-4a9c-9680-64cc1ebc2157\" (UID: \"874390a6-68da-4a9c-9680-64cc1ebc2157\") " Nov 03 19:01:39.032856 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.032818 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-openshift-ca" pod "874390a6-68da-4a9c-9680-64cc1ebc2157" (UID: "874390a6-68da-4a9c-9680-64cc1ebc2157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:39.034694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.034666 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-xks-ca" pod "874390a6-68da-4a9c-9680-64cc1ebc2157" (UID: "874390a6-68da-4a9c-9680-64cc1ebc2157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:39.034694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.034739 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874390a6-68da-4a9c-9680-64cc1ebc2157-kube-api-access-4hwxb" pod "874390a6-68da-4a9c-9680-64cc1ebc2157" (UID: "874390a6-68da-4a9c-9680-64cc1ebc2157"). InnerVolumeSpecName "kube-api-access-4hwxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:39.134294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.134239 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:39.134497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.134480 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwxb\" (UniqueName: \"kubernetes.io/projected/874390a6-68da-4a9c-9680-64cc1ebc2157-kube-api-access-4hwxb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:39.134561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.134553 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874390a6-68da-4a9c-9680-64cc1ebc2157-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:39.481815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.481722 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:01:39.481815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.481789 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 19:01:39.705382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.705354 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" Nov 03 19:01:39.705827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.705387 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj" event={"ID":"874390a6-68da-4a9c-9680-64cc1ebc2157","Type":"ContainerDied","Data":"3c280813ee52d22de188c23f29ef945003b4b3dd60fd1ab7d54bc58e6779acb0"} Nov 03 19:01:39.705827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.705485 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c280813ee52d22de188c23f29ef945003b4b3dd60fd1ab7d54bc58e6779acb0" Nov 03 19:01:39.710412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.710381 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" event={"ID":"e056629f-76b0-49a7-b632-48a5729bf91f","Type":"ContainerStarted","Data":"77fa4fbc9001ea8edec8c86ccf36610520a56af453f36246688d078dabedb86c"} Nov 03 19:01:39.710586 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.710427 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" event={"ID":"e056629f-76b0-49a7-b632-48a5729bf91f","Type":"ContainerStarted","Data":"b0649d5a59f1b011fc345a06281637fb4ba0717ff2995717cd8ca1288ad324a9"} Nov 03 19:01:39.837100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:39.836996 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" podStartSLOduration=1.836982995 podStartE2EDuration="1.836982995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:39.833139376 +0000 UTC m=+1429.316135314" watchObservedRunningTime="2026-11-03 19:01:39.836982995 +0000 UTC m=+1429.319978909" Nov 03 19:01:40.481922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:40.481889 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:01:40.482090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:40.481943 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 19:01:40.721214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:40.721181 4948 generic.go:356] "Generic (PLEG): container finished" podID="e056629f-76b0-49a7-b632-48a5729bf91f" containerID="77fa4fbc9001ea8edec8c86ccf36610520a56af453f36246688d078dabedb86c" exitCode=0 Nov 03 19:01:40.721638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:40.721264 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" event={"ID":"e056629f-76b0-49a7-b632-48a5729bf91f","Type":"ContainerDied","Data":"77fa4fbc9001ea8edec8c86ccf36610520a56af453f36246688d078dabedb86c"} Nov 03 19:01:41.482592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:41.482550 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:01:41.482784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:41.482616 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 19:01:41.935148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:41.935122 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:42.000794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.000762 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-openshift-ca\") pod \"e056629f-76b0-49a7-b632-48a5729bf91f\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " Nov 03 19:01:42.000957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.000935 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e056629f-76b0-49a7-b632-48a5729bf91f-kube-api-access-8bknj\" (UniqueName: \"kubernetes.io/projected/e056629f-76b0-49a7-b632-48a5729bf91f-kube-api-access-8bknj\") pod \"e056629f-76b0-49a7-b632-48a5729bf91f\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " Nov 03 19:01:42.001039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.001011 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-xks-ca\") pod \"e056629f-76b0-49a7-b632-48a5729bf91f\" (UID: \"e056629f-76b0-49a7-b632-48a5729bf91f\") " Nov 03 19:01:42.001165 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.001143 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-openshift-ca" pod "e056629f-76b0-49a7-b632-48a5729bf91f" (UID: "e056629f-76b0-49a7-b632-48a5729bf91f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:42.001577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.001552 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:42.003283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.003253 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-xks-ca" pod "e056629f-76b0-49a7-b632-48a5729bf91f" (UID: "e056629f-76b0-49a7-b632-48a5729bf91f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:42.004400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.004374 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e056629f-76b0-49a7-b632-48a5729bf91f-kube-api-access-8bknj" pod "e056629f-76b0-49a7-b632-48a5729bf91f" (UID: "e056629f-76b0-49a7-b632-48a5729bf91f"). InnerVolumeSpecName "kube-api-access-8bknj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:42.103069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.102950 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e056629f-76b0-49a7-b632-48a5729bf91f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:42.103069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.102988 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bknj\" (UniqueName: \"kubernetes.io/projected/e056629f-76b0-49a7-b632-48a5729bf91f-kube-api-access-8bknj\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:42.481938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.481899 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:01:42.482148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.481955 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 19:01:42.744260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.744180 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" Nov 03 19:01:42.744260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.744203 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c" event={"ID":"e056629f-76b0-49a7-b632-48a5729bf91f","Type":"ContainerDied","Data":"b0649d5a59f1b011fc345a06281637fb4ba0717ff2995717cd8ca1288ad324a9"} Nov 03 19:01:42.744446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:42.744297 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0649d5a59f1b011fc345a06281637fb4ba0717ff2995717cd8ca1288ad324a9" Nov 03 19:01:43.056417 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.056316 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t"] Nov 03 19:01:43.061642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.061614 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a4a8f2-fp52t"] Nov 03 19:01:43.112938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.112902 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w"] Nov 03 19:01:43.114009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.113979 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e056629f-76b0-49a7-b632-48a5729bf91f" containerName="revoke-keys" Nov 03 19:01:43.114009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.114008 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e056629f-76b0-49a7-b632-48a5729bf91f" containerName="revoke-keys" Nov 03 19:01:43.114366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.114151 4948 cpu_manager.go:414] "containerMap: removing container" podUID="874390a6-68da-4a9c-9680-64cc1ebc2157" containerName="revoke-keys" Nov 03 19:01:43.114366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.114166 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874390a6-68da-4a9c-9680-64cc1ebc2157" containerName="revoke-keys" Nov 03 19:01:43.114572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.114469 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e056629f-76b0-49a7-b632-48a5729bf91f" containerName="revoke-keys" Nov 03 19:01:43.114572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.114490 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="874390a6-68da-4a9c-9680-64cc1ebc2157" containerName="revoke-keys" Nov 03 19:01:43.119359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.119335 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.169681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.169647 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w"] Nov 03 19:01:43.172869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.172843 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:01:43.173219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.173199 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:01:43.173307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.173202 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:01:43.173561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.173543 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:01:43.219008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.218966 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvkz\" (UniqueName: \"kubernetes.io/projected/18a8e82a-4018-4b2d-8555-3583e266313f-kube-api-access-pwvkz\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.219197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.219089 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.219197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.219156 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.320539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.320452 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvkz\" (UniqueName: \"kubernetes.io/projected/18a8e82a-4018-4b2d-8555-3583e266313f-kube-api-access-pwvkz\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.320539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.320511 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.320739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.320556 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.321314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.321289 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.322910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.322888 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.383346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.383314 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvkz\" (UniqueName: \"kubernetes.io/projected/18a8e82a-4018-4b2d-8555-3583e266313f-kube-api-access-pwvkz\") pod \"maas-api-revoke-sub-8e7fd6b9-m9k2w\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.435342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.435310 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:43.488470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.488441 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:43.489115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.489084 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:43.490160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.490138 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:01:43.829756 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:43.825409 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a8e82a_4018_4b2d_8555_3583e266313f.slice/crio-aafd701697a929bfcb1b46c8f73a75889c32fa01c31eee7b7027b27eb12fb929 WatchSource:0}: Error finding container aafd701697a929bfcb1b46c8f73a75889c32fa01c31eee7b7027b27eb12fb929: Status 404 returned error can't find the container with id aafd701697a929bfcb1b46c8f73a75889c32fa01c31eee7b7027b27eb12fb929 Nov 03 19:01:43.829756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:43.827653 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w"] Nov 03 19:01:44.055689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.055661 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6"] Nov 03 19:01:44.061222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.061191 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5a369b-kpff6"] Nov 03 19:01:44.066076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.066055 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m"] Nov 03 19:01:44.070645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.070594 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4038e619-2bn9m"] Nov 03 19:01:44.758545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.758494 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332895aa-8af7-48b5-a350-aa18b4b300b3" path="/var/lib/kubelet/pods/332895aa-8af7-48b5-a350-aa18b4b300b3/volumes" Nov 03 19:01:44.759489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.759464 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44411451-2cd4-4e63-ac95-dc9249b1d2d9" path="/var/lib/kubelet/pods/44411451-2cd4-4e63-ac95-dc9249b1d2d9/volumes" Nov 03 19:01:44.760326 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.760307 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ce1ba0-6c27-41a9-899c-737a2083f7bc" path="/var/lib/kubelet/pods/83ce1ba0-6c27-41a9-899c-737a2083f7bc/volumes" Nov 03 19:01:44.775943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.775913 4948 generic.go:356] "Generic (PLEG): container finished" podID="18a8e82a-4018-4b2d-8555-3583e266313f" containerID="6fa43d726c1224f119130c7550532f461ad3074466647b4bc0af081447e1539c" exitCode=0 Nov 03 19:01:44.776101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.775998 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" event={"ID":"18a8e82a-4018-4b2d-8555-3583e266313f","Type":"ContainerDied","Data":"6fa43d726c1224f119130c7550532f461ad3074466647b4bc0af081447e1539c"} Nov 03 19:01:44.776101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:44.776049 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" event={"ID":"18a8e82a-4018-4b2d-8555-3583e266313f","Type":"ContainerStarted","Data":"aafd701697a929bfcb1b46c8f73a75889c32fa01c31eee7b7027b27eb12fb929"} Nov 03 19:01:45.960573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:45.960546 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:46.066956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.066872 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-openshift-ca\") pod \"18a8e82a-4018-4b2d-8555-3583e266313f\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " Nov 03 19:01:46.066956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.066918 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-xks-ca\") pod \"18a8e82a-4018-4b2d-8555-3583e266313f\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " Nov 03 19:01:46.067137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.067071 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a8e82a-4018-4b2d-8555-3583e266313f-kube-api-access-pwvkz\" (UniqueName: \"kubernetes.io/projected/18a8e82a-4018-4b2d-8555-3583e266313f-kube-api-access-pwvkz\") pod \"18a8e82a-4018-4b2d-8555-3583e266313f\" (UID: \"18a8e82a-4018-4b2d-8555-3583e266313f\") " Nov 03 19:01:46.067313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.067292 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-openshift-ca" pod "18a8e82a-4018-4b2d-8555-3583e266313f" (UID: "18a8e82a-4018-4b2d-8555-3583e266313f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:46.067685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.067647 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:46.069211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.069176 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-xks-ca" pod "18a8e82a-4018-4b2d-8555-3583e266313f" (UID: "18a8e82a-4018-4b2d-8555-3583e266313f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:46.069344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.069320 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a8e82a-4018-4b2d-8555-3583e266313f-kube-api-access-pwvkz" pod "18a8e82a-4018-4b2d-8555-3583e266313f" (UID: "18a8e82a-4018-4b2d-8555-3583e266313f"). InnerVolumeSpecName "kube-api-access-pwvkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:46.168876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.168840 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a8e82a-4018-4b2d-8555-3583e266313f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:46.168876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.168870 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvkz\" (UniqueName: \"kubernetes.io/projected/18a8e82a-4018-4b2d-8555-3583e266313f-kube-api-access-pwvkz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:46.796166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.796139 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" Nov 03 19:01:46.796308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.796169 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w" event={"ID":"18a8e82a-4018-4b2d-8555-3583e266313f","Type":"ContainerDied","Data":"aafd701697a929bfcb1b46c8f73a75889c32fa01c31eee7b7027b27eb12fb929"} Nov 03 19:01:46.796308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:46.796214 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aafd701697a929bfcb1b46c8f73a75889c32fa01c31eee7b7027b27eb12fb929" Nov 03 19:01:47.055130 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:47.055041 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk"] Nov 03 19:01:47.063169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:47.063134 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ef9ce-ftcfk"] Nov 03 19:01:48.169294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.169257 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v"] Nov 03 19:01:48.170049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.170017 4948 cpu_manager.go:414] "containerMap: removing container" podUID="18a8e82a-4018-4b2d-8555-3583e266313f" containerName="revoke-keys" Nov 03 19:01:48.170049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.170036 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a8e82a-4018-4b2d-8555-3583e266313f" containerName="revoke-keys" Nov 03 19:01:48.170226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.170211 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a8e82a-4018-4b2d-8555-3583e266313f" containerName="revoke-keys" Nov 03 19:01:48.174496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.174475 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.228126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.228090 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v"] Nov 03 19:01:48.228290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.228116 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:01:48.272951 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.272912 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:01:48.273142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.273082 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:01:48.273208 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.273189 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:01:48.298613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.298583 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.298771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.298747 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.298815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.298804 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xx4\" (UniqueName: \"kubernetes.io/projected/eefe3c15-f5cc-4845-bb2d-a5b41220b954-kube-api-access-f9xx4\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.400729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.400653 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.400939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.400829 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.400939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.400917 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xx4\" (UniqueName: \"kubernetes.io/projected/eefe3c15-f5cc-4845-bb2d-a5b41220b954-kube-api-access-f9xx4\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.401660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.401628 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.403242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.403221 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.463188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.463162 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xx4\" (UniqueName: \"kubernetes.io/projected/eefe3c15-f5cc-4845-bb2d-a5b41220b954-kube-api-access-f9xx4\") pod \"maas-api-revoke-sub-5f09ce36-f526v\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.503720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.498934 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:48.756146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.756063 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc27eb40-4ab7-4b1d-8611-9373b4b4410a" path="/var/lib/kubelet/pods/fc27eb40-4ab7-4b1d-8611-9373b4b4410a/volumes" Nov 03 19:01:48.951373 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:48.950672 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefe3c15_f5cc_4845_bb2d_a5b41220b954.slice/crio-838e123d6798fe9f02977074233c409f3983263c0026aede703a3c7e42869cc7 WatchSource:0}: Error finding container 838e123d6798fe9f02977074233c409f3983263c0026aede703a3c7e42869cc7: Status 404 returned error can't find the container with id 838e123d6798fe9f02977074233c409f3983263c0026aede703a3c7e42869cc7 Nov 03 19:01:48.952227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:48.952198 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v"] Nov 03 19:01:49.194238 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.194207 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84"] Nov 03 19:01:49.200940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.200922 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.260865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.260834 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84"] Nov 03 19:01:49.319000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.318965 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbft5\" (UniqueName: \"kubernetes.io/projected/13a07937-1758-4c74-bd78-9032f9606925-kube-api-access-pbft5\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.319168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.319074 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a07937-1758-4c74-bd78-9032f9606925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.319231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.319197 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a07937-1758-4c74-bd78-9032f9606925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.352485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.352454 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6"] Nov 03 19:01:49.357523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.357503 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.418627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.418546 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6"] Nov 03 19:01:49.421263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.421236 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbft5\" (UniqueName: \"kubernetes.io/projected/13a07937-1758-4c74-bd78-9032f9606925-kube-api-access-pbft5\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.421397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.421299 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a07937-1758-4c74-bd78-9032f9606925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.421460 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.421408 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a07937-1758-4c74-bd78-9032f9606925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.422773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.422689 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a07937-1758-4c74-bd78-9032f9606925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.424740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.424682 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a07937-1758-4c74-bd78-9032f9606925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.490995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.490962 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbft5\" (UniqueName: \"kubernetes.io/projected/13a07937-1758-4c74-bd78-9032f9606925-kube-api-access-pbft5\") pod \"maas-api-revoke-sub-74fee69a-z6p84\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.516989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.516959 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:49.522747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.522720 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.522854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.522816 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e356e-72df-49be-9213-ac45383f3950-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.522927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.522853 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.522990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.522971 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.523038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.522998 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjqhq\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-kube-api-access-cjqhq\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.523038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.523027 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.523171 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.523158 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.523210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.523188 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e356e-72df-49be-9213-ac45383f3950-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.523334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.523313 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.610175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.610139 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks"] Nov 03 19:01:49.621083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.621056 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.625417 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625387 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625434 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e356e-72df-49be-9213-ac45383f3950-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625547 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625598 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625673 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e356e-72df-49be-9213-ac45383f3950-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625739 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625820 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625850 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjqhq\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-kube-api-access-cjqhq\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.625980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.625891 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.626172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.626102 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.627577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.626367 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.627577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.626606 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.627577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.627171 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e356e-72df-49be-9213-ac45383f3950-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.628013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.627986 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.630104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.630074 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.630365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.630346 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e356e-72df-49be-9213-ac45383f3950-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.671626 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.671555 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks"] Nov 03 19:01:49.688069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.688030 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.689072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.689035 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjqhq\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-kube-api-access-cjqhq\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:49.727486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.727446 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8w9\" (UniqueName: \"kubernetes.io/projected/3a08a1a7-f03d-4932-a580-073ce53525c1-kube-api-access-vv8w9\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.727656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.727640 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.727860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.727845 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.829510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.829481 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.829736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.829601 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8w9\" (UniqueName: \"kubernetes.io/projected/3a08a1a7-f03d-4932-a580-073ce53525c1-kube-api-access-vv8w9\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.829810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.829774 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.830298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.830272 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.831456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.831427 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" event={"ID":"eefe3c15-f5cc-4845-bb2d-a5b41220b954","Type":"ContainerStarted","Data":"4d98e9f65eb171175c16eeb87f3990169dbef83a9955d59f5edbdf6fec597433"} Nov 03 19:01:49.831573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.831493 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" event={"ID":"eefe3c15-f5cc-4845-bb2d-a5b41220b954","Type":"ContainerStarted","Data":"838e123d6798fe9f02977074233c409f3983263c0026aede703a3c7e42869cc7"} Nov 03 19:01:49.832284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.832263 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.891845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.891814 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8w9\" (UniqueName: \"kubernetes.io/projected/3a08a1a7-f03d-4932-a580-073ce53525c1-kube-api-access-vv8w9\") pod \"maas-api-revoke-sub-12d8a519-mgfks\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.913962 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:49.913933 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a07937_1758_4c74_bd78_9032f9606925.slice/crio-afbff78a09699f420e89d86e394ad4fb54cb36acc98cde0d61efd2026de8c68a WatchSource:0}: Error finding container afbff78a09699f420e89d86e394ad4fb54cb36acc98cde0d61efd2026de8c68a: Status 404 returned error can't find the container with id afbff78a09699f420e89d86e394ad4fb54cb36acc98cde0d61efd2026de8c68a Nov 03 19:01:49.916880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.916795 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84"] Nov 03 19:01:49.942215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.942189 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:49.952503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.952444 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" podStartSLOduration=1.952425222 podStartE2EDuration="1.952425222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:49.948369531 +0000 UTC m=+1439.431365439" watchObservedRunningTime="2026-11-03 19:01:49.952425222 +0000 UTC m=+1439.435421137" Nov 03 19:01:49.975534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:49.975497 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:50.081234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.080785 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8"] Nov 03 19:01:50.089568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.089535 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.134609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.134558 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8"] Nov 03 19:01:50.235290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235258 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235316 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235408 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235528 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235556 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235584 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a659e245-b62d-4586-ac00-205ba64f6842-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235616 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235641 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a659e245-b62d-4586-ac00-205ba64f6842-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.235884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.235666 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz487\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-kube-api-access-sz487\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.335251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.335220 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks"] Nov 03 19:01:50.337138 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337107 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337179 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337218 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337297 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337378 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337322 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337378 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337347 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a659e245-b62d-4586-ac00-205ba64f6842-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337394 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337438 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a659e245-b62d-4586-ac00-205ba64f6842-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337615 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337673 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz487\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-kube-api-access-sz487\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337737 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337834 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.337999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.337975 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.338230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.338211 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a659e245-b62d-4586-ac00-205ba64f6842-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.351370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.351343 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.351659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.351639 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a659e245-b62d-4586-ac00-205ba64f6842-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.353294 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:50.353266 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a08a1a7_f03d_4932_a580_073ce53525c1.slice/crio-ca3db60c68f221b621d46d14c6007d0042f7dc0fd916277277dd4a02a1f062df WatchSource:0}: Error finding container ca3db60c68f221b621d46d14c6007d0042f7dc0fd916277277dd4a02a1f062df: Status 404 returned error can't find the container with id ca3db60c68f221b621d46d14c6007d0042f7dc0fd916277277dd4a02a1f062df Nov 03 19:01:50.390925 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:50.390813 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20e356e_72df_49be_9213_ac45383f3950.slice/crio-8e898ecc5e9d864ad244c2b6fa650925d314a67bb19f832013542c1dba35ce47 WatchSource:0}: Error finding container 8e898ecc5e9d864ad244c2b6fa650925d314a67bb19f832013542c1dba35ce47: Status 404 returned error can't find the container with id 8e898ecc5e9d864ad244c2b6fa650925d314a67bb19f832013542c1dba35ce47 Nov 03 19:01:50.392606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.392571 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:50.392714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.392655 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:50.392786 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.392716 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:50.394640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.394612 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6"] Nov 03 19:01:50.400683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.400657 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz487\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-kube-api-access-sz487\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.401123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.401101 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.410563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.410542 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:50.824344 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:01:50.824311 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda659e245_b62d_4586_ac00_205ba64f6842.slice/crio-189f182dafe23b1a9d87a956a8f7ec637d812c2134ca5739060178df78273dde WatchSource:0}: Error finding container 189f182dafe23b1a9d87a956a8f7ec637d812c2134ca5739060178df78273dde: Status 404 returned error can't find the container with id 189f182dafe23b1a9d87a956a8f7ec637d812c2134ca5739060178df78273dde Nov 03 19:01:50.828517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.828426 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8"] Nov 03 19:01:50.830902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.830863 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:50.831011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.830946 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:50.831011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.830988 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:01:50.848684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.848648 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" event={"ID":"b20e356e-72df-49be-9213-ac45383f3950","Type":"ContainerStarted","Data":"9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9"} Nov 03 19:01:50.848864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.848732 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" event={"ID":"b20e356e-72df-49be-9213-ac45383f3950","Type":"ContainerStarted","Data":"8e898ecc5e9d864ad244c2b6fa650925d314a67bb19f832013542c1dba35ce47"} Nov 03 19:01:50.851392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.851361 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" event={"ID":"3a08a1a7-f03d-4932-a580-073ce53525c1","Type":"ContainerStarted","Data":"ca3db60c68f221b621d46d14c6007d0042f7dc0fd916277277dd4a02a1f062df"} Nov 03 19:01:50.853745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.853675 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" event={"ID":"a659e245-b62d-4586-ac00-205ba64f6842","Type":"ContainerStarted","Data":"189f182dafe23b1a9d87a956a8f7ec637d812c2134ca5739060178df78273dde"} Nov 03 19:01:50.858793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.858765 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" event={"ID":"13a07937-1758-4c74-bd78-9032f9606925","Type":"ContainerStarted","Data":"4b4a2cff90fb14bd982b5a087723c0d1293b2c61d4e8440195a54ada5892dbca"} Nov 03 19:01:50.858904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.858816 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" event={"ID":"13a07937-1758-4c74-bd78-9032f9606925","Type":"ContainerStarted","Data":"afbff78a09699f420e89d86e394ad4fb54cb36acc98cde0d61efd2026de8c68a"} Nov 03 19:01:50.862289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.862256 4948 generic.go:356] "Generic (PLEG): container finished" podID="eefe3c15-f5cc-4845-bb2d-a5b41220b954" containerID="4d98e9f65eb171175c16eeb87f3990169dbef83a9955d59f5edbdf6fec597433" exitCode=0 Nov 03 19:01:50.862795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.862542 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" event={"ID":"eefe3c15-f5cc-4845-bb2d-a5b41220b954","Type":"ContainerDied","Data":"4d98e9f65eb171175c16eeb87f3990169dbef83a9955d59f5edbdf6fec597433"} Nov 03 19:01:50.977069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.976989 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:01:50.986531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:50.986470 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podStartSLOduration=1.9864447570000001 podStartE2EDuration="1.986444757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:50.972746527 +0000 UTC m=+1440.455742450" watchObservedRunningTime="2026-11-03 19:01:50.986444757 +0000 UTC m=+1440.469440672" Nov 03 19:01:51.212726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.211876 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" podStartSLOduration=2.211856931 podStartE2EDuration="2.211856931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:51.208788546 +0000 UTC m=+1440.691784460" watchObservedRunningTime="2026-11-03 19:01:51.211856931 +0000 UTC m=+1440.694852845" Nov 03 19:01:51.919807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.919463 4948 generic.go:356] "Generic (PLEG): container finished" podID="3a08a1a7-f03d-4932-a580-073ce53525c1" containerID="afc71c9e5b4f75cfd50450aa771740254167c330fbb84d72b50b1b5df93c9df4" exitCode=0 Nov 03 19:01:51.919807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.919519 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" event={"ID":"3a08a1a7-f03d-4932-a580-073ce53525c1","Type":"ContainerDied","Data":"afc71c9e5b4f75cfd50450aa771740254167c330fbb84d72b50b1b5df93c9df4"} Nov 03 19:01:51.925908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.925874 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" event={"ID":"a659e245-b62d-4586-ac00-205ba64f6842","Type":"ContainerStarted","Data":"e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91"} Nov 03 19:01:51.928402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.928377 4948 generic.go:356] "Generic (PLEG): container finished" podID="13a07937-1758-4c74-bd78-9032f9606925" containerID="4b4a2cff90fb14bd982b5a087723c0d1293b2c61d4e8440195a54ada5892dbca" exitCode=0 Nov 03 19:01:51.929047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.929023 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" event={"ID":"13a07937-1758-4c74-bd78-9032f9606925","Type":"ContainerDied","Data":"4b4a2cff90fb14bd982b5a087723c0d1293b2c61d4e8440195a54ada5892dbca"} Nov 03 19:01:51.983722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.981500 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:01:51.983722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:51.981570 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:01:52.212755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.210880 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podStartSLOduration=2.210858145 podStartE2EDuration="2.210858145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:52.206257505 +0000 UTC m=+1441.689253420" watchObservedRunningTime="2026-11-03 19:01:52.210858145 +0000 UTC m=+1441.693854059" Nov 03 19:01:52.325267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.325229 4948 scope.go:122] "RemoveContainer" containerID="2c3859f58454c2ada2680cc4ebf8fb8e695d33f3693fc65c50dccbe0b1b413fd" Nov 03 19:01:52.331559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.331530 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:52.344619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.344599 4948 scope.go:122] "RemoveContainer" containerID="fcd445ed247ae5937f83731e602d2cd5e6f8743079dcc40cb699155e5652584e" Nov 03 19:01:52.413752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.412030 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:01:52.421166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.420888 4948 scope.go:122] "RemoveContainer" containerID="37a5686019a51b7d2a13f77dd3a2cce3f0a4af43726b8d23c0c62233252b8300" Nov 03 19:01:52.421166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.421060 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:01:52.421166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.421103 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:01:52.426518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.426493 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-openshift-ca\") pod \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " Nov 03 19:01:52.426636 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.426561 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefe3c15-f5cc-4845-bb2d-a5b41220b954-kube-api-access-f9xx4\" (UniqueName: \"kubernetes.io/projected/eefe3c15-f5cc-4845-bb2d-a5b41220b954-kube-api-access-f9xx4\") pod \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " Nov 03 19:01:52.426636 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.426620 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-xks-ca\") pod \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\" (UID: \"eefe3c15-f5cc-4845-bb2d-a5b41220b954\") " Nov 03 19:01:52.427786 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.427758 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-openshift-ca" pod "eefe3c15-f5cc-4845-bb2d-a5b41220b954" (UID: "eefe3c15-f5cc-4845-bb2d-a5b41220b954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:52.429352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.429332 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-xks-ca" pod "eefe3c15-f5cc-4845-bb2d-a5b41220b954" (UID: "eefe3c15-f5cc-4845-bb2d-a5b41220b954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:52.431221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.430271 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefe3c15-f5cc-4845-bb2d-a5b41220b954-kube-api-access-f9xx4" pod "eefe3c15-f5cc-4845-bb2d-a5b41220b954" (UID: "eefe3c15-f5cc-4845-bb2d-a5b41220b954"). InnerVolumeSpecName "kube-api-access-f9xx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:52.443965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.443901 4948 scope.go:122] "RemoveContainer" containerID="1aed2d476c7c58e28b3feed82ff6d6c62db1384bd1b56f395da8b717d39085cf" Nov 03 19:01:52.529153 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.529068 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:52.529378 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.529361 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9xx4\" (UniqueName: \"kubernetes.io/projected/eefe3c15-f5cc-4845-bb2d-a5b41220b954-kube-api-access-f9xx4\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:52.529469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.529459 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3c15-f5cc-4845-bb2d-a5b41220b954-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:52.938634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.938606 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" Nov 03 19:01:52.939104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.939086 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v" event={"ID":"eefe3c15-f5cc-4845-bb2d-a5b41220b954","Type":"ContainerDied","Data":"838e123d6798fe9f02977074233c409f3983263c0026aede703a3c7e42869cc7"} Nov 03 19:01:52.939151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.939122 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838e123d6798fe9f02977074233c409f3983263c0026aede703a3c7e42869cc7" Nov 03 19:01:52.979160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.979104 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:01:52.979371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:52.979185 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:01:53.336255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.335885 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:53.417792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.412245 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:01:53.417792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.412292 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:01:53.510687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.476979 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-xks-ca\") pod \"3a08a1a7-f03d-4932-a580-073ce53525c1\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " Nov 03 19:01:53.510687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.477099 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a08a1a7-f03d-4932-a580-073ce53525c1-kube-api-access-vv8w9\" (UniqueName: \"kubernetes.io/projected/3a08a1a7-f03d-4932-a580-073ce53525c1-kube-api-access-vv8w9\") pod \"3a08a1a7-f03d-4932-a580-073ce53525c1\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " Nov 03 19:01:53.510687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.477174 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-openshift-ca\") pod \"3a08a1a7-f03d-4932-a580-073ce53525c1\" (UID: \"3a08a1a7-f03d-4932-a580-073ce53525c1\") " Nov 03 19:01:53.525742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.525683 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-openshift-ca" pod "3a08a1a7-f03d-4932-a580-073ce53525c1" (UID: "3a08a1a7-f03d-4932-a580-073ce53525c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:53.599725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.587098 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:53.608387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.607923 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:53.617048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.616974 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-xks-ca" pod "3a08a1a7-f03d-4932-a580-073ce53525c1" (UID: "3a08a1a7-f03d-4932-a580-073ce53525c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:53.680822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.667903 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a08a1a7-f03d-4932-a580-073ce53525c1-kube-api-access-vv8w9" pod "3a08a1a7-f03d-4932-a580-073ce53525c1" (UID: "3a08a1a7-f03d-4932-a580-073ce53525c1"). InnerVolumeSpecName "kube-api-access-vv8w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:53.701270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.695160 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a07937-1758-4c74-bd78-9032f9606925-kube-api-access-pbft5\" (UniqueName: \"kubernetes.io/projected/13a07937-1758-4c74-bd78-9032f9606925-kube-api-access-pbft5\") pod \"13a07937-1758-4c74-bd78-9032f9606925\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " Nov 03 19:01:53.701270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.695237 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13a07937-1758-4c74-bd78-9032f9606925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a07937-1758-4c74-bd78-9032f9606925-maas-api-xks-ca\") pod \"13a07937-1758-4c74-bd78-9032f9606925\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " Nov 03 19:01:53.701270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.695305 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a07937-1758-4c74-bd78-9032f9606925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a07937-1758-4c74-bd78-9032f9606925-maas-api-openshift-ca\") pod \"13a07937-1758-4c74-bd78-9032f9606925\" (UID: \"13a07937-1758-4c74-bd78-9032f9606925\") " Nov 03 19:01:53.701270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.696049 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a08a1a7-f03d-4932-a580-073ce53525c1-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:53.701270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.696074 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv8w9\" (UniqueName: \"kubernetes.io/projected/3a08a1a7-f03d-4932-a580-073ce53525c1-kube-api-access-vv8w9\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:53.739020 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.738972 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a07937-1758-4c74-bd78-9032f9606925-maas-api-openshift-ca" pod "13a07937-1758-4c74-bd78-9032f9606925" (UID: "13a07937-1758-4c74-bd78-9032f9606925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:01:53.741452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.741349 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a07937-1758-4c74-bd78-9032f9606925-kube-api-access-pbft5" pod "13a07937-1758-4c74-bd78-9032f9606925" (UID: "13a07937-1758-4c74-bd78-9032f9606925"). InnerVolumeSpecName "kube-api-access-pbft5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:01:53.743219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.743178 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a07937-1758-4c74-bd78-9032f9606925-maas-api-xks-ca" pod "13a07937-1758-4c74-bd78-9032f9606925" (UID: "13a07937-1758-4c74-bd78-9032f9606925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:01:53.826736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.826007 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbft5\" (UniqueName: \"kubernetes.io/projected/13a07937-1758-4c74-bd78-9032f9606925-kube-api-access-pbft5\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:53.826736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.826067 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a07937-1758-4c74-bd78-9032f9606925-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:53.826736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.826087 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a07937-1758-4c74-bd78-9032f9606925-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:01:53.949628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.949594 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" Nov 03 19:01:53.950144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.949914 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks" event={"ID":"3a08a1a7-f03d-4932-a580-073ce53525c1","Type":"ContainerDied","Data":"ca3db60c68f221b621d46d14c6007d0042f7dc0fd916277277dd4a02a1f062df"} Nov 03 19:01:53.950144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.949977 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3db60c68f221b621d46d14c6007d0042f7dc0fd916277277dd4a02a1f062df" Nov 03 19:01:53.969831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.958895 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" Nov 03 19:01:53.986720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.980053 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84" event={"ID":"13a07937-1758-4c74-bd78-9032f9606925","Type":"ContainerDied","Data":"afbff78a09699f420e89d86e394ad4fb54cb36acc98cde0d61efd2026de8c68a"} Nov 03 19:01:53.986720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:53.980117 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbff78a09699f420e89d86e394ad4fb54cb36acc98cde0d61efd2026de8c68a" Nov 03 19:01:54.020961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:54.020232 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:01:54.020961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:54.020295 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:01:54.415531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:54.411122 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:01:54.415531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:54.411187 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:01:54.996858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:54.996817 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:01:54.997437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:54.997400 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:01:55.059277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:55.059236 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d"] Nov 03 19:01:55.064694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:55.064650 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5de52ce-xfs4d"] Nov 03 19:01:55.411778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:55.411397 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:01:55.411778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:55.411481 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:01:55.978749 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:55.977817 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:01:55.978749 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:55.977885 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:01:56.097724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.096771 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp"] Nov 03 19:01:56.122505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.122470 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b078fad8-689qp"] Nov 03 19:01:56.412227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.412126 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:01:56.412227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.412198 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:01:56.779770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.775278 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bef0166-1c95-4f78-88e3-e098e1343d1e" path="/var/lib/kubelet/pods/2bef0166-1c95-4f78-88e3-e098e1343d1e/volumes" Nov 03 19:01:56.779770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.776051 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd1ba67-b3d2-461a-af8a-d6d030af1a27" path="/var/lib/kubelet/pods/2fd1ba67-b3d2-461a-af8a-d6d030af1a27/volumes" Nov 03 19:01:56.984731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.984096 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:01:56.984731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:56.984173 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:01:57.411356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:57.411317 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:01:57.411830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:57.411377 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:01:57.978160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:57.978126 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:01:57.978444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:57.978410 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:01:58.416777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:58.413208 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:01:58.416777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:58.413277 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:01:59.016723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.016360 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62"] Nov 03 19:01:59.017727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017452 4948 cpu_manager.go:414] "containerMap: removing container" podUID="13a07937-1758-4c74-bd78-9032f9606925" containerName="revoke-keys" Nov 03 19:01:59.017727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017478 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a07937-1758-4c74-bd78-9032f9606925" containerName="revoke-keys" Nov 03 19:01:59.017727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017514 4948 cpu_manager.go:414] "containerMap: removing container" podUID="eefe3c15-f5cc-4845-bb2d-a5b41220b954" containerName="revoke-keys" Nov 03 19:01:59.017727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017524 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefe3c15-f5cc-4845-bb2d-a5b41220b954" containerName="revoke-keys" Nov 03 19:01:59.017727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017550 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3a08a1a7-f03d-4932-a580-073ce53525c1" containerName="revoke-keys" Nov 03 19:01:59.017727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017559 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a08a1a7-f03d-4932-a580-073ce53525c1" containerName="revoke-keys" Nov 03 19:01:59.018144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017758 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefe3c15-f5cc-4845-bb2d-a5b41220b954" containerName="revoke-keys" Nov 03 19:01:59.018144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017775 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a08a1a7-f03d-4932-a580-073ce53525c1" containerName="revoke-keys" Nov 03 19:01:59.018144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.017799 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a07937-1758-4c74-bd78-9032f9606925" containerName="revoke-keys" Nov 03 19:01:59.028728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.028035 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.075723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.074538 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62"] Nov 03 19:01:59.112756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.096213 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:01:59.112756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.096316 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:01:59.112756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.096343 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:01:59.112756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.097023 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:01:59.240477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.239660 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.240477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.239749 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvz74\" (UniqueName: \"kubernetes.io/projected/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-kube-api-access-nvz74\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.240477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.239800 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.387617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.387498 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.393908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.393878 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvz74\" (UniqueName: \"kubernetes.io/projected/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-kube-api-access-nvz74\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.394176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.394158 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.400470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.400340 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.433859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.433826 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.489727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.489201 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvz74\" (UniqueName: \"kubernetes.io/projected/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-kube-api-access-nvz74\") pod \"maas-api-revoke-sub-baa5f68d-5mf62\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:01:59.675088 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:01:59.673955 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:02:00.026994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.026950 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:00.027366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.027332 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.078794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.078758 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq"] Nov 03 19:02:00.139795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.139757 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq"] Nov 03 19:02:00.160452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.160378 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ae7e51-gmhwq"] Nov 03 19:02:00.193738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.183664 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e61e90-vsgmq"] Nov 03 19:02:00.706784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.653871 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:00.706784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.653941 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.706784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.663806 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.816914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:00.762536 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.049590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.048797 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:01.049590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.048881 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.160725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.156726 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63" path="/var/lib/kubelet/pods/49d4eb00-ff37-4ca4-b3ae-3bb445c9cf63/volumes" Nov 03 19:02:01.160725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.157448 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9c5ad5-8c1e-4962-96c7-15f77188bd48" path="/var/lib/kubelet/pods/8d9c5ad5-8c1e-4962-96c7-15f77188bd48/volumes" Nov 03 19:02:01.283781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.281089 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548"] Nov 03 19:02:01.306933 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.306875 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:01.469450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.469397 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.585769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.563040 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:01.605599 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:01.605566 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf9b5ac_92fe_4a21_b6b1_290321440d0c.slice/crio-fa93ccbc988b54ad100f86ecf9eb4a13fa4ea875e2f4f1d45e1f9ab14b951a02 WatchSource:0}: Error finding container fa93ccbc988b54ad100f86ecf9eb4a13fa4ea875e2f4f1d45e1f9ab14b951a02: Status 404 returned error can't find the container with id fa93ccbc988b54ad100f86ecf9eb4a13fa4ea875e2f4f1d45e1f9ab14b951a02 Nov 03 19:02:01.705438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.705338 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548"] Nov 03 19:02:01.840199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.775349 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:01.840199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.775670 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:01.840199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.775910 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:02:01.840199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.777616 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:01.840199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.777654 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.840199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.778106 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:02:01.858343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.857879 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:02:01.886323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.886296 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:02:01.896152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.895785 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.962766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.956872 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntrh\" (UniqueName: \"kubernetes.io/projected/fc51c215-341b-4565-9dbb-5c5493f51937-kube-api-access-9ntrh\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:01.962766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.956955 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:01.962766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.957210 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:01.962766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.959062 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 19:02:01.993243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:01.989632 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.023728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.014050 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.077958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.063418 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:02.228726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.202154 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntrh\" (UniqueName: \"kubernetes.io/projected/fc51c215-341b-4565-9dbb-5c5493f51937-kube-api-access-9ntrh\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:02.228726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.205779 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:02.236855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.231084 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:02.331782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.300015 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:02.331782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.300102 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.331782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.311079 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.459740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.459287 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz"] Nov 03 19:02:02.614032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.613138 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:02.721726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.711115 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:02:02.857783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.857475 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:02.858285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.857812 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:02.870735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.861175 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:02:02.870735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.869034 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:02:02.917726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.916879 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:02.917726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.916966 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.938726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.935673 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntrh\" (UniqueName: \"kubernetes.io/projected/fc51c215-341b-4565-9dbb-5c5493f51937-kube-api-access-9ntrh\") pod \"maas-api-revoke-sub-0d8d3e1d-t2548\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:02.938726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:02.936040 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:03.016138 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.016004 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:02:03.046587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.016750 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:03.129299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.129203 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 03 19:02:03.425457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.425327 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:03.595913 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.539306 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Nov 03 19:02:03.758870 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.721719 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:03.804444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.804407 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:03.891752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.866742 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:02:04.015230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.897387 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:04.041368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.974722 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:04.042295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.975416 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:04.067725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:03.984264 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:04.068131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.009842 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:04.068332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.068300 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:04.077722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.033571 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:04.400999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.375133 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:04.400999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.375235 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzkv\" (UniqueName: \"kubernetes.io/projected/daa225ba-ef5e-470f-9cd2-c3911e843663-kube-api-access-kqzkv\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:04.400999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.375305 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:04.526758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.501356 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:04.545069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.537347 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:04.559724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.550794 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzkv\" (UniqueName: \"kubernetes.io/projected/daa225ba-ef5e-470f-9cd2-c3911e843663-kube-api-access-kqzkv\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:04.559724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.550917 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:04.622012 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:02:04.621970 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:02:04.651846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.625150 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:02:04.654182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.654146 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" event={"ID":"eaf9b5ac-92fe-4a21-b6b1-290321440d0c","Type":"ContainerStarted","Data":"fa93ccbc988b54ad100f86ecf9eb4a13fa4ea875e2f4f1d45e1f9ab14b951a02"} Nov 03 19:02:04.657726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.654377 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz"] Nov 03 19:02:04.657983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.657964 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62"] Nov 03 19:02:04.658172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.658152 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:02:04.672726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.658308 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:02:04.673037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.673014 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:02:04.673960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.673941 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:02:04.674785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.674769 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:02:04.723728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.718824 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:02:04.723728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.718862 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:02:04.723728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.628797 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:04.744733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.724890 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.017747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.996410 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:05.017747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:04.996479 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.076726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.066675 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:02:05.077179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.077135 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.077648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.077618 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:05.078175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.078152 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:05.078762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.078737 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.119742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.093051 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.149240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.149201 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:05.160881 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.160838 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.668600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.667005 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:05.668600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.667065 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.668600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.667777 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ktxrg" podUID="c659d570-af36-4aca-9532-5dc9e24ef622" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.744753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.719603 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" podUID="621ec304-b99e-4a18-b966-dd7e1e0c73b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.744753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.722618 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:05.744753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.724443 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:05.744753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.725080 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.744753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.741277 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" podUID="621ec304-b99e-4a18-b966-dd7e1e0c73b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:05.854595 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:02:05.854565 4948 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.078s" Nov 03 19:02:05.856894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.856875 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:02:05.880126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.878378 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzkv\" (UniqueName: \"kubernetes.io/projected/daa225ba-ef5e-470f-9cd2-c3911e843663-kube-api-access-kqzkv\") pod \"maas-api-revoke-sub-b8b0b2e2-6gfnz\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:05.933756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:05.933564 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4"] Nov 03 19:02:06.028305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.028271 4948 generic.go:356] "Generic (PLEG): container finished" podID="eaf9b5ac-92fe-4a21-b6b1-290321440d0c" containerID="56445504831ea0d5a09cce9b6307c04a251f6d0b93146c9e15c1ea3096fe971d" exitCode=0 Nov 03 19:02:06.030787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.030752 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" event={"ID":"eaf9b5ac-92fe-4a21-b6b1-290321440d0c","Type":"ContainerDied","Data":"56445504831ea0d5a09cce9b6307c04a251f6d0b93146c9e15c1ea3096fe971d"} Nov 03 19:02:06.059276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.059242 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e56449-lthf4"] Nov 03 19:02:06.069814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.066011 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:02:06.079764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.077792 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:06.079933 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.079804 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:06.088783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.086251 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:06.088783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.086892 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:06.107071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.102901 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:06.110242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.110215 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:02:06.157622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.148223 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:06.179266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.173853 4948 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:06.179266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.174205 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:06.199878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.199783 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548"] Nov 03 19:02:06.309301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.309268 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:02:06.466564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.464886 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:02:06.503723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.501320 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:02:06.525724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.522293 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:02:06.527157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.526790 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:02:06.548066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.548034 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:02:06.585490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.585459 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:02:06.610050 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.610016 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:02:06.760784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:06.760754 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d07c08e-d01a-45b1-83b9-4e6032930224" path="/var/lib/kubelet/pods/0d07c08e-d01a-45b1-83b9-4e6032930224/volumes" Nov 03 19:02:07.095666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.095637 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" event={"ID":"fc51c215-341b-4565-9dbb-5c5493f51937","Type":"ContainerStarted","Data":"367083fab54c6da23838d2bc3c2fd88cb530e3d2b70f6163837b89eec2f372aa"} Nov 03 19:02:07.096261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.096221 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" event={"ID":"fc51c215-341b-4565-9dbb-5c5493f51937","Type":"ContainerStarted","Data":"786c9738ffe7602cca00461dea7f235d7524d37674cba8600736e5371a0d009d"} Nov 03 19:02:07.096412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.096397 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz"] Nov 03 19:02:07.096602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.096589 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:02:07.511886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.511858 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:02:07.672616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.672521 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-xks-ca\") pod \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " Nov 03 19:02:07.672837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.672787 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-kube-api-access-nvz74\" (UniqueName: \"kubernetes.io/projected/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-kube-api-access-nvz74\") pod \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " Nov 03 19:02:07.672837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.672823 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-openshift-ca\") pod \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\" (UID: \"eaf9b5ac-92fe-4a21-b6b1-290321440d0c\") " Nov 03 19:02:07.673585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.673526 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-openshift-ca" pod "eaf9b5ac-92fe-4a21-b6b1-290321440d0c" (UID: "eaf9b5ac-92fe-4a21-b6b1-290321440d0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:07.675210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.675176 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-xks-ca" pod "eaf9b5ac-92fe-4a21-b6b1-290321440d0c" (UID: "eaf9b5ac-92fe-4a21-b6b1-290321440d0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:07.675558 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.675525 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-kube-api-access-nvz74" pod "eaf9b5ac-92fe-4a21-b6b1-290321440d0c" (UID: "eaf9b5ac-92fe-4a21-b6b1-290321440d0c"). InnerVolumeSpecName "kube-api-access-nvz74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:07.775298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.775261 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvz74\" (UniqueName: \"kubernetes.io/projected/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-kube-api-access-nvz74\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:07.775298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.775296 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:07.775553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:07.775311 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf9b5ac-92fe-4a21-b6b1-290321440d0c-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:08.055689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.055656 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" event={"ID":"eaf9b5ac-92fe-4a21-b6b1-290321440d0c","Type":"ContainerDied","Data":"fa93ccbc988b54ad100f86ecf9eb4a13fa4ea875e2f4f1d45e1f9ab14b951a02"} Nov 03 19:02:08.055986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.055958 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa93ccbc988b54ad100f86ecf9eb4a13fa4ea875e2f4f1d45e1f9ab14b951a02" Nov 03 19:02:08.056332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.056310 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62" Nov 03 19:02:08.058246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.058057 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" event={"ID":"daa225ba-ef5e-470f-9cd2-c3911e843663","Type":"ContainerStarted","Data":"eb0e974cde947ac33ed566cb3c91771e65db4b1413622d7b99df7aec1618fc12"} Nov 03 19:02:08.058246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.058206 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" event={"ID":"daa225ba-ef5e-470f-9cd2-c3911e843663","Type":"ContainerStarted","Data":"47df246755c098411e3b9c373368e81d4615a08191bb35386012dfc45ffbefd7"} Nov 03 19:02:08.060172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.060148 4948 generic.go:356] "Generic (PLEG): container finished" podID="fc51c215-341b-4565-9dbb-5c5493f51937" containerID="367083fab54c6da23838d2bc3c2fd88cb530e3d2b70f6163837b89eec2f372aa" exitCode=0 Nov 03 19:02:08.060291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.060217 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" event={"ID":"fc51c215-341b-4565-9dbb-5c5493f51937","Type":"ContainerDied","Data":"367083fab54c6da23838d2bc3c2fd88cb530e3d2b70f6163837b89eec2f372aa"} Nov 03 19:02:08.584902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:08.584842 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" podStartSLOduration=7.584822857 podStartE2EDuration="7.584822857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:08.584065033 +0000 UTC m=+1458.067060948" watchObservedRunningTime="2026-11-03 19:02:08.584822857 +0000 UTC m=+1458.067818773" Nov 03 19:02:09.071941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.071900 4948 generic.go:356] "Generic (PLEG): container finished" podID="daa225ba-ef5e-470f-9cd2-c3911e843663" containerID="eb0e974cde947ac33ed566cb3c91771e65db4b1413622d7b99df7aec1618fc12" exitCode=0 Nov 03 19:02:09.072120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.071981 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" event={"ID":"daa225ba-ef5e-470f-9cd2-c3911e843663","Type":"ContainerDied","Data":"eb0e974cde947ac33ed566cb3c91771e65db4b1413622d7b99df7aec1618fc12"} Nov 03 19:02:09.246492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.246466 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:09.326019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.325937 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc51c215-341b-4565-9dbb-5c5493f51937-kube-api-access-9ntrh\" (UniqueName: \"kubernetes.io/projected/fc51c215-341b-4565-9dbb-5c5493f51937-kube-api-access-9ntrh\") pod \"fc51c215-341b-4565-9dbb-5c5493f51937\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " Nov 03 19:02:09.326186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.326166 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-xks-ca\") pod \"fc51c215-341b-4565-9dbb-5c5493f51937\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " Nov 03 19:02:09.326318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.326303 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-openshift-ca\") pod \"fc51c215-341b-4565-9dbb-5c5493f51937\" (UID: \"fc51c215-341b-4565-9dbb-5c5493f51937\") " Nov 03 19:02:09.326673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.326643 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-openshift-ca" pod "fc51c215-341b-4565-9dbb-5c5493f51937" (UID: "fc51c215-341b-4565-9dbb-5c5493f51937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:09.326846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.326829 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:09.328155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.328131 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-xks-ca" pod "fc51c215-341b-4565-9dbb-5c5493f51937" (UID: "fc51c215-341b-4565-9dbb-5c5493f51937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:09.328240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.328176 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc51c215-341b-4565-9dbb-5c5493f51937-kube-api-access-9ntrh" pod "fc51c215-341b-4565-9dbb-5c5493f51937" (UID: "fc51c215-341b-4565-9dbb-5c5493f51937"). InnerVolumeSpecName "kube-api-access-9ntrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:09.428084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.428052 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ntrh\" (UniqueName: \"kubernetes.io/projected/fc51c215-341b-4565-9dbb-5c5493f51937-kube-api-access-9ntrh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:09.428084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.428082 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc51c215-341b-4565-9dbb-5c5493f51937-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:09.705045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.705011 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb"] Nov 03 19:02:09.705749 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.705732 4948 cpu_manager.go:414] "containerMap: removing container" podUID="eaf9b5ac-92fe-4a21-b6b1-290321440d0c" containerName="revoke-keys" Nov 03 19:02:09.705817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.705749 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf9b5ac-92fe-4a21-b6b1-290321440d0c" containerName="revoke-keys" Nov 03 19:02:09.705817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.705762 4948 cpu_manager.go:414] "containerMap: removing container" podUID="fc51c215-341b-4565-9dbb-5c5493f51937" containerName="revoke-keys" Nov 03 19:02:09.705817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.705767 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc51c215-341b-4565-9dbb-5c5493f51937" containerName="revoke-keys" Nov 03 19:02:09.705970 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.705886 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf9b5ac-92fe-4a21-b6b1-290321440d0c" containerName="revoke-keys" Nov 03 19:02:09.705970 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.705894 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc51c215-341b-4565-9dbb-5c5493f51937" containerName="revoke-keys" Nov 03 19:02:09.709834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.709818 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.731616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.731586 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.731777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.731642 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhrfc\" (UniqueName: \"kubernetes.io/projected/41375853-6b74-44b4-8b99-0cd15393831f-kube-api-access-qhrfc\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.731855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.731833 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.768126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.768071 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb"] Nov 03 19:02:09.834026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.833988 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.834266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.834247 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.834315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.834297 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhrfc\" (UniqueName: \"kubernetes.io/projected/41375853-6b74-44b4-8b99-0cd15393831f-kube-api-access-qhrfc\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.835117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.835083 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.836631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.836602 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:09.899322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:09.899292 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhrfc\" (UniqueName: \"kubernetes.io/projected/41375853-6b74-44b4-8b99-0cd15393831f-kube-api-access-qhrfc\") pod \"maas-api-revoke-sub-0e43a950-rlctb\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:10.031259 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.031166 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:10.209912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.193403 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" Nov 03 19:02:10.209912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.209766 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548" event={"ID":"fc51c215-341b-4565-9dbb-5c5493f51937","Type":"ContainerDied","Data":"786c9738ffe7602cca00461dea7f235d7524d37674cba8600736e5371a0d009d"} Nov 03 19:02:10.209912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.209846 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="786c9738ffe7602cca00461dea7f235d7524d37674cba8600736e5371a0d009d" Nov 03 19:02:10.693720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.692873 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt"] Nov 03 19:02:10.730483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.730397 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:10.774031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.773998 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt"] Nov 03 19:02:10.774272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.774256 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g"] Nov 03 19:02:10.784467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.784437 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g"] Nov 03 19:02:10.784609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.784556 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:10.786024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.785968 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-90774e61\"" Nov 03 19:02:10.786886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.786362 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-90774e61-dockercfg-ssfzp\"" Nov 03 19:02:10.884014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.883983 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:10.884196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.884117 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8f77\" (UniqueName: \"kubernetes.io/projected/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-kube-api-access-x8f77\") pod \"payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:10.884242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.884205 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjg7w\" (UniqueName: \"kubernetes.io/projected/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-kube-api-access-wjg7w\") pod \"payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:10.884344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.884329 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:10.921144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.921106 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:10.985390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.985316 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-openshift-ca\") pod \"daa225ba-ef5e-470f-9cd2-c3911e843663\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " Nov 03 19:02:10.985520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.985466 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-xks-ca\") pod \"daa225ba-ef5e-470f-9cd2-c3911e843663\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " Nov 03 19:02:10.985520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.985493 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa225ba-ef5e-470f-9cd2-c3911e843663-kube-api-access-kqzkv\" (UniqueName: \"kubernetes.io/projected/daa225ba-ef5e-470f-9cd2-c3911e843663-kube-api-access-kqzkv\") pod \"daa225ba-ef5e-470f-9cd2-c3911e843663\" (UID: \"daa225ba-ef5e-470f-9cd2-c3911e843663\") " Nov 03 19:02:10.985678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.985659 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8f77\" (UniqueName: \"kubernetes.io/projected/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-kube-api-access-x8f77\") pod \"payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:10.985806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.985787 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjg7w\" (UniqueName: \"kubernetes.io/projected/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-kube-api-access-wjg7w\") pod \"payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:10.985870 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.985792 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-openshift-ca" pod "daa225ba-ef5e-470f-9cd2-c3911e843663" (UID: "daa225ba-ef5e-470f-9cd2-c3911e843663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:10.985980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.985957 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:10.986114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.986092 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:10.986247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.986225 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:10.987197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.987089 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:10.987197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.987147 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:10.987946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.987923 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa225ba-ef5e-470f-9cd2-c3911e843663-kube-api-access-kqzkv" pod "daa225ba-ef5e-470f-9cd2-c3911e843663" (UID: "daa225ba-ef5e-470f-9cd2-c3911e843663"). InnerVolumeSpecName "kube-api-access-kqzkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:10.988199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:10.988181 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-xks-ca" pod "daa225ba-ef5e-470f-9cd2-c3911e843663" (UID: "daa225ba-ef5e-470f-9cd2-c3911e843663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:11.008462 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:11.008435 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41375853_6b74_44b4_8b99_0cd15393831f.slice/crio-89360a23c36fcd829fd4864e19c21cded617ad0660ce14d4e66a044e3074a85c WatchSource:0}: Error finding container 89360a23c36fcd829fd4864e19c21cded617ad0660ce14d4e66a044e3074a85c: Status 404 returned error can't find the container with id 89360a23c36fcd829fd4864e19c21cded617ad0660ce14d4e66a044e3074a85c Nov 03 19:02:11.009917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.009895 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb"] Nov 03 19:02:11.050132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.050103 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjg7w\" (UniqueName: \"kubernetes.io/projected/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-kube-api-access-wjg7w\") pod \"payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:11.050257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.050158 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8f77\" (UniqueName: \"kubernetes.io/projected/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-kube-api-access-x8f77\") pod \"payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:11.059335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.059310 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9"] Nov 03 19:02:11.067929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.067906 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87"] Nov 03 19:02:11.073506 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.073483 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:11.076105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.076084 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722beb94-t62s9"] Nov 03 19:02:11.084148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.084119 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5161eb69-fnv87"] Nov 03 19:02:11.087617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.087589 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa225ba-ef5e-470f-9cd2-c3911e843663-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:11.087758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.087622 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqzkv\" (UniqueName: \"kubernetes.io/projected/daa225ba-ef5e-470f-9cd2-c3911e843663-kube-api-access-kqzkv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:11.108297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.108266 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:11.218572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.218532 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" event={"ID":"daa225ba-ef5e-470f-9cd2-c3911e843663","Type":"ContainerDied","Data":"47df246755c098411e3b9c373368e81d4615a08191bb35386012dfc45ffbefd7"} Nov 03 19:02:11.218676 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.218611 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47df246755c098411e3b9c373368e81d4615a08191bb35386012dfc45ffbefd7" Nov 03 19:02:11.218755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.218726 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz" Nov 03 19:02:11.220960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.220901 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" event={"ID":"41375853-6b74-44b4-8b99-0cd15393831f","Type":"ContainerStarted","Data":"89360a23c36fcd829fd4864e19c21cded617ad0660ce14d4e66a044e3074a85c"} Nov 03 19:02:11.539793 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:11.539691 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2d1c8b_8fe9_4e6e_80c6_926d59697f0c.slice/crio-009f0ce9b7d490eafd0b55aee5ccfa657981e7598d87d075d529cd4241226afa WatchSource:0}: Error finding container 009f0ce9b7d490eafd0b55aee5ccfa657981e7598d87d075d529cd4241226afa: Status 404 returned error can't find the container with id 009f0ce9b7d490eafd0b55aee5ccfa657981e7598d87d075d529cd4241226afa Nov 03 19:02:11.542820 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.542796 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt"] Nov 03 19:02:11.593567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:11.593531 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g"] Nov 03 19:02:11.594489 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:11.594458 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b51d6f9_dbec_4fa0_a0b3_072746a6485d.slice/crio-a66d3010fbf223016adf981cbe3336be86803ab69a276488ed932691ca822828 WatchSource:0}: Error finding container a66d3010fbf223016adf981cbe3336be86803ab69a276488ed932691ca822828: Status 404 returned error can't find the container with id a66d3010fbf223016adf981cbe3336be86803ab69a276488ed932691ca822828 Nov 03 19:02:12.302278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.302244 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz"] Nov 03 19:02:12.304085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.304011 4948 cpu_manager.go:414] "containerMap: removing container" podUID="daa225ba-ef5e-470f-9cd2-c3911e843663" containerName="revoke-keys" Nov 03 19:02:12.304241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.304225 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa225ba-ef5e-470f-9cd2-c3911e843663" containerName="revoke-keys" Nov 03 19:02:12.304588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.304572 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa225ba-ef5e-470f-9cd2-c3911e843663" containerName="revoke-keys" Nov 03 19:02:12.318993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.318959 4948 generic.go:356] "Generic (PLEG): container finished" podID="41375853-6b74-44b4-8b99-0cd15393831f" containerID="c95913aea9c9bb82eea3704c00bb0c4c98547f92c52603f898078a6145d58cf4" exitCode=0 Nov 03 19:02:12.326518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.326488 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:12.326804 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.326771 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" event={"ID":"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c","Type":"ContainerStarted","Data":"4bd5a8bc6f2294a43ad7fc3772caff4c82a32fe0c658beb8e1b5419611098988"} Nov 03 19:02:12.327033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.326989 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" event={"ID":"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c","Type":"ContainerStarted","Data":"009f0ce9b7d490eafd0b55aee5ccfa657981e7598d87d075d529cd4241226afa"} Nov 03 19:02:12.327165 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.327151 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:12.327272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.327254 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" event={"ID":"41375853-6b74-44b4-8b99-0cd15393831f","Type":"ContainerDied","Data":"c95913aea9c9bb82eea3704c00bb0c4c98547f92c52603f898078a6145d58cf4"} Nov 03 19:02:12.327389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.327373 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" event={"ID":"4b51d6f9-dbec-4fa0-a0b3-072746a6485d","Type":"ContainerStarted","Data":"76457c297db8c85ddaa9cd9f762b317678719d5a89fb2b1faff101472511e29f"} Nov 03 19:02:12.327512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.327495 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" event={"ID":"4b51d6f9-dbec-4fa0-a0b3-072746a6485d","Type":"ContainerStarted","Data":"a66d3010fbf223016adf981cbe3336be86803ab69a276488ed932691ca822828"} Nov 03 19:02:12.327837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.327817 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.437231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.437185 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-token\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.437568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.437535 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-certs\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.437845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.437825 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-envoy\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.438021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.438003 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-podinfo\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.438170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.438154 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istiod-ca-cert\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.438647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.438617 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-socket\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.446510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.438896 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brsx9\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-kube-api-access-brsx9\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.460361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.453765 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6f198bfb-openshift-default-dockercfg-vc87j\"" Nov 03 19:02:12.460361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.456997 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-data\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.460361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.457097 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-credential-socket\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.460361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.460178 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz"] Nov 03 19:02:12.469738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.464832 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" podStartSLOduration=2.464815063 podStartE2EDuration="2.464815063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:12.451646008 +0000 UTC m=+1461.934641925" watchObservedRunningTime="2026-11-03 19:02:12.464815063 +0000 UTC m=+1461.947810978" Nov 03 19:02:12.559118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559025 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-token\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559072 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-certs\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559101 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-envoy\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559131 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-podinfo\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559168 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istiod-ca-cert\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559273 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-socket\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559300 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brsx9\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-kube-api-access-brsx9\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559336 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-data\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559436 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-credential-socket\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.559770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.559745 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-socket\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.560069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.560040 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-credential-socket\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.561012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.560503 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istiod-ca-cert\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.561162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.560830 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-data\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.561252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.560871 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-certs\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.562111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.562072 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-envoy\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.563905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.563883 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-podinfo\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.572779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.572721 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" podStartSLOduration=2.572684638 podStartE2EDuration="2.572684638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:12.570070044 +0000 UTC m=+1462.053065958" watchObservedRunningTime="2026-11-03 19:02:12.572684638 +0000 UTC m=+1462.055680553" Nov 03 19:02:12.623385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.623344 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-token\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.623625 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.623597 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brsx9\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-kube-api-access-brsx9\") pod \"e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.765229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.764950 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:12.842822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.832043 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ada7914-3437-4c4e-b39f-04610941e91f" path="/var/lib/kubelet/pods/4ada7914-3437-4c4e-b39f-04610941e91f/volumes" Nov 03 19:02:12.855549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:12.843383 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641376a9-6fb0-480e-bee1-f48c41b4ac89" path="/var/lib/kubelet/pods/641376a9-6fb0-480e-bee1-f48c41b4ac89/volumes" Nov 03 19:02:13.288593 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:13.288559 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf002b4ad_77d9_4c88_b2bd_d49102a8ef7a.slice/crio-838e0e57a20bb89218aa4e864a6b469c4de9979547bbe73afe47c09836d2221e WatchSource:0}: Error finding container 838e0e57a20bb89218aa4e864a6b469c4de9979547bbe73afe47c09836d2221e: Status 404 returned error can't find the container with id 838e0e57a20bb89218aa4e864a6b469c4de9979547bbe73afe47c09836d2221e Nov 03 19:02:13.295638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.292046 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz"] Nov 03 19:02:13.295638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.295577 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:13.295792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.295685 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:13.295792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.295747 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:13.335352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.335316 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" event={"ID":"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a","Type":"ContainerStarted","Data":"838e0e57a20bb89218aa4e864a6b469c4de9979547bbe73afe47c09836d2221e"} Nov 03 19:02:13.539301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.539239 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:13.577582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.577550 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41375853-6b74-44b4-8b99-0cd15393831f-kube-api-access-qhrfc\" (UniqueName: \"kubernetes.io/projected/41375853-6b74-44b4-8b99-0cd15393831f-kube-api-access-qhrfc\") pod \"41375853-6b74-44b4-8b99-0cd15393831f\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " Nov 03 19:02:13.577794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.577736 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-openshift-ca\") pod \"41375853-6b74-44b4-8b99-0cd15393831f\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " Nov 03 19:02:13.577917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.577898 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-xks-ca\") pod \"41375853-6b74-44b4-8b99-0cd15393831f\" (UID: \"41375853-6b74-44b4-8b99-0cd15393831f\") " Nov 03 19:02:13.578103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.578072 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-openshift-ca" pod "41375853-6b74-44b4-8b99-0cd15393831f" (UID: "41375853-6b74-44b4-8b99-0cd15393831f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:13.578494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.578474 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:13.579873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.579849 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41375853-6b74-44b4-8b99-0cd15393831f-kube-api-access-qhrfc" pod "41375853-6b74-44b4-8b99-0cd15393831f" (UID: "41375853-6b74-44b4-8b99-0cd15393831f"). InnerVolumeSpecName "kube-api-access-qhrfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:13.579873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.579857 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-xks-ca" pod "41375853-6b74-44b4-8b99-0cd15393831f" (UID: "41375853-6b74-44b4-8b99-0cd15393831f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:13.679472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.679435 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41375853-6b74-44b4-8b99-0cd15393831f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:13.679472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:13.679467 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhrfc\" (UniqueName: \"kubernetes.io/projected/41375853-6b74-44b4-8b99-0cd15393831f-kube-api-access-qhrfc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:14.346934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:14.346902 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" Nov 03 19:02:14.347564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:14.346902 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb" event={"ID":"41375853-6b74-44b4-8b99-0cd15393831f","Type":"ContainerDied","Data":"89360a23c36fcd829fd4864e19c21cded617ad0660ce14d4e66a044e3074a85c"} Nov 03 19:02:14.347564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:14.347058 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89360a23c36fcd829fd4864e19c21cded617ad0660ce14d4e66a044e3074a85c" Nov 03 19:02:14.348988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:14.348952 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" event={"ID":"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a","Type":"ContainerStarted","Data":"dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350"} Nov 03 19:02:14.475052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:14.474947 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podStartSLOduration=2.4749315689999998 podStartE2EDuration="2.474931569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:14.471972418 +0000 UTC m=+1463.954968324" watchObservedRunningTime="2026-11-03 19:02:14.474931569 +0000 UTC m=+1463.957927483" Nov 03 19:02:14.766246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:14.766176 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:14.771753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:14.771691 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:15.385713 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:15.385666 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:15.386617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:15.386597 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:02:18.068812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.068778 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt"] Nov 03 19:02:18.120226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.120188 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a65e4d8-fcjbt"] Nov 03 19:02:18.125855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.125829 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg"] Nov 03 19:02:18.126512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.126497 4948 cpu_manager.go:414] "containerMap: removing container" podUID="41375853-6b74-44b4-8b99-0cd15393831f" containerName="revoke-keys" Nov 03 19:02:18.126561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.126513 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41375853-6b74-44b4-8b99-0cd15393831f" containerName="revoke-keys" Nov 03 19:02:18.126671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.126660 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="41375853-6b74-44b4-8b99-0cd15393831f" containerName="revoke-keys" Nov 03 19:02:18.130770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.130750 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.172713 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.172676 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg"] Nov 03 19:02:18.184536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.184509 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:02:18.184762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.184598 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:02:18.184762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.184603 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:02:18.184762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.184660 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:02:18.197733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.197676 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.197876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.197825 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.197998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.197980 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqh2p\" (UniqueName: \"kubernetes.io/projected/8c2a8780-28fd-4595-a86b-0be48526c486-kube-api-access-pqh2p\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.299534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.299504 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.299752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.299622 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqh2p\" (UniqueName: \"kubernetes.io/projected/8c2a8780-28fd-4595-a86b-0be48526c486-kube-api-access-pqh2p\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.299842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.299777 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.300401 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.300369 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.302167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.302147 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.363199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.363124 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqh2p\" (UniqueName: \"kubernetes.io/projected/8c2a8780-28fd-4595-a86b-0be48526c486-kube-api-access-pqh2p\") pod \"maas-api-revoke-sub-cb7e57a2-fzbgg\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.445775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.445739 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:18.755914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.755880 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e" path="/var/lib/kubelet/pods/7a318f7b-c0b6-4f1e-8a4e-cbf6287ecb6e/volumes" Nov 03 19:02:18.829728 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:18.829670 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2a8780_28fd_4595_a86b_0be48526c486.slice/crio-fe9375c30fe4cf2961a0330fd33fd15852ea89cc98839b03593d66bf58544235 WatchSource:0}: Error finding container fe9375c30fe4cf2961a0330fd33fd15852ea89cc98839b03593d66bf58544235: Status 404 returned error can't find the container with id fe9375c30fe4cf2961a0330fd33fd15852ea89cc98839b03593d66bf58544235 Nov 03 19:02:18.833727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:18.833680 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg"] Nov 03 19:02:19.429597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:19.429498 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" event={"ID":"8c2a8780-28fd-4595-a86b-0be48526c486","Type":"ContainerStarted","Data":"83d7b9f3ea32a2914d848f142e51259915dd3434ffdd0d6f25637fe8bdc2912e"} Nov 03 19:02:19.430102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:19.429615 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" event={"ID":"8c2a8780-28fd-4595-a86b-0be48526c486","Type":"ContainerStarted","Data":"fe9375c30fe4cf2961a0330fd33fd15852ea89cc98839b03593d66bf58544235"} Nov 03 19:02:19.556449 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:19.556390 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" podStartSLOduration=1.556375331 podStartE2EDuration="1.556375331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:19.551424919 +0000 UTC m=+1469.034420896" watchObservedRunningTime="2026-11-03 19:02:19.556375331 +0000 UTC m=+1469.039371245" Nov 03 19:02:19.966754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:19.966718 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7"] Nov 03 19:02:19.971335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:19.971318 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.025384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.025350 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7"] Nov 03 19:02:20.127319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.127226 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4df7583-55ad-4641-96dc-49b643910558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.127482 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.127390 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprgf\" (UniqueName: \"kubernetes.io/projected/c4df7583-55ad-4641-96dc-49b643910558-kube-api-access-pprgf\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.127544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.127526 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4df7583-55ad-4641-96dc-49b643910558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.228617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.228579 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4df7583-55ad-4641-96dc-49b643910558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.228793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.228728 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4df7583-55ad-4641-96dc-49b643910558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.228882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.228834 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprgf\" (UniqueName: \"kubernetes.io/projected/c4df7583-55ad-4641-96dc-49b643910558-kube-api-access-pprgf\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.229331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.229309 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4df7583-55ad-4641-96dc-49b643910558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.231259 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.231232 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4df7583-55ad-4641-96dc-49b643910558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.291285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.291257 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprgf\" (UniqueName: \"kubernetes.io/projected/c4df7583-55ad-4641-96dc-49b643910558-kube-api-access-pprgf\") pod \"maas-api-revoke-sub-6c878b2f-4s5c7\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.440355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.440321 4948 generic.go:356] "Generic (PLEG): container finished" podID="8c2a8780-28fd-4595-a86b-0be48526c486" containerID="83d7b9f3ea32a2914d848f142e51259915dd3434ffdd0d6f25637fe8bdc2912e" exitCode=0 Nov 03 19:02:20.440850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.440407 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" event={"ID":"8c2a8780-28fd-4595-a86b-0be48526c486","Type":"ContainerDied","Data":"83d7b9f3ea32a2914d848f142e51259915dd3434ffdd0d6f25637fe8bdc2912e"} Nov 03 19:02:20.586318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.586279 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:20.981844 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:20.981810 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4df7583_55ad_4641_96dc_49b643910558.slice/crio-1a1b6d5e7aa19e76d4946c984d2898e603735ef9782bec0e31619cabc7deff4e WatchSource:0}: Error finding container 1a1b6d5e7aa19e76d4946c984d2898e603735ef9782bec0e31619cabc7deff4e: Status 404 returned error can't find the container with id 1a1b6d5e7aa19e76d4946c984d2898e603735ef9782bec0e31619cabc7deff4e Nov 03 19:02:20.983838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:20.983715 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7"] Nov 03 19:02:21.472218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.470878 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x"] Nov 03 19:02:21.498172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.498132 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:21.502423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.502383 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" event={"ID":"c4df7583-55ad-4641-96dc-49b643910558","Type":"ContainerStarted","Data":"1a1b6d5e7aa19e76d4946c984d2898e603735ef9782bec0e31619cabc7deff4e"} Nov 03 19:02:21.529789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.529751 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x"] Nov 03 19:02:21.579123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.579100 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6f198bfb-dockercfg-67mvw\"" Nov 03 19:02:21.581582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.581505 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6f198bfb\"" Nov 03 19:02:21.587017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.586976 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljds\" (UniqueName: \"kubernetes.io/projected/4eec5248-4944-46a7-bff9-5da030569f0f-kube-api-access-fljds\") pod \"payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:21.587150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.587066 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eec5248-4944-46a7-bff9-5da030569f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:21.710767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.708047 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljds\" (UniqueName: \"kubernetes.io/projected/4eec5248-4944-46a7-bff9-5da030569f0f-kube-api-access-fljds\") pod \"payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:21.710767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.708146 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eec5248-4944-46a7-bff9-5da030569f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:21.710767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.709201 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eec5248-4944-46a7-bff9-5da030569f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:21.827835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.827799 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljds\" (UniqueName: \"kubernetes.io/projected/4eec5248-4944-46a7-bff9-5da030569f0f-kube-api-access-fljds\") pod \"payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:21.870562 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.867606 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" podStartSLOduration=2.867586104 podStartE2EDuration="2.867586104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:21.799243351 +0000 UTC m=+1471.282239276" watchObservedRunningTime="2026-11-03 19:02:21.867586104 +0000 UTC m=+1471.350582020" Nov 03 19:02:21.947925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:21.947901 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:22.029369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.029285 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2a8780-28fd-4595-a86b-0be48526c486-kube-api-access-pqh2p\" (UniqueName: \"kubernetes.io/projected/8c2a8780-28fd-4595-a86b-0be48526c486-kube-api-access-pqh2p\") pod \"8c2a8780-28fd-4595-a86b-0be48526c486\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " Nov 03 19:02:22.029521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.029504 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-xks-ca\") pod \"8c2a8780-28fd-4595-a86b-0be48526c486\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " Nov 03 19:02:22.029572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.029527 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-openshift-ca\") pod \"8c2a8780-28fd-4595-a86b-0be48526c486\" (UID: \"8c2a8780-28fd-4595-a86b-0be48526c486\") " Nov 03 19:02:22.029989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.029956 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-openshift-ca" pod "8c2a8780-28fd-4595-a86b-0be48526c486" (UID: "8c2a8780-28fd-4595-a86b-0be48526c486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:22.031381 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.031355 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a8780-28fd-4595-a86b-0be48526c486-kube-api-access-pqh2p" pod "8c2a8780-28fd-4595-a86b-0be48526c486" (UID: "8c2a8780-28fd-4595-a86b-0be48526c486"). InnerVolumeSpecName "kube-api-access-pqh2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:22.031497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.031475 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-xks-ca" pod "8c2a8780-28fd-4595-a86b-0be48526c486" (UID: "8c2a8780-28fd-4595-a86b-0be48526c486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:22.123717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.123470 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:22.131488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.131451 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:22.131488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.131486 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a8780-28fd-4595-a86b-0be48526c486-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:22.131640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.131503 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqh2p\" (UniqueName: \"kubernetes.io/projected/8c2a8780-28fd-4595-a86b-0be48526c486-kube-api-access-pqh2p\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:22.492711 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.492675 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj"] Nov 03 19:02:22.493468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.493449 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8c2a8780-28fd-4595-a86b-0be48526c486" containerName="revoke-keys" Nov 03 19:02:22.493525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.493468 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2a8780-28fd-4595-a86b-0be48526c486" containerName="revoke-keys" Nov 03 19:02:22.493593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.493583 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2a8780-28fd-4595-a86b-0be48526c486" containerName="revoke-keys" Nov 03 19:02:22.498417 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.498398 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.509180 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:22.509154 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eec5248_4944_46a7_bff9_5da030569f0f.slice/crio-76ed12db59eb77db990fc2d7e4311ff3bf2c3122512cfdf1f15897d821f59ebe WatchSource:0}: Error finding container 76ed12db59eb77db990fc2d7e4311ff3bf2c3122512cfdf1f15897d821f59ebe: Status 404 returned error can't find the container with id 76ed12db59eb77db990fc2d7e4311ff3bf2c3122512cfdf1f15897d821f59ebe Nov 03 19:02:22.510602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.510572 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x"] Nov 03 19:02:22.526787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.526760 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" Nov 03 19:02:22.526787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.526764 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg" event={"ID":"8c2a8780-28fd-4595-a86b-0be48526c486","Type":"ContainerDied","Data":"fe9375c30fe4cf2961a0330fd33fd15852ea89cc98839b03593d66bf58544235"} Nov 03 19:02:22.526943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.526855 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe9375c30fe4cf2961a0330fd33fd15852ea89cc98839b03593d66bf58544235" Nov 03 19:02:22.529300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.529272 4948 generic.go:356] "Generic (PLEG): container finished" podID="c4df7583-55ad-4641-96dc-49b643910558" containerID="f50e2c746fdb6de8643ee02a0814dc0ff59dd5f3d5f3ffd1fde9ab83cb90d516" exitCode=0 Nov 03 19:02:22.529402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.529351 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" event={"ID":"c4df7583-55ad-4641-96dc-49b643910558","Type":"ContainerDied","Data":"f50e2c746fdb6de8643ee02a0814dc0ff59dd5f3d5f3ffd1fde9ab83cb90d516"} Nov 03 19:02:22.552587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.552560 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj"] Nov 03 19:02:22.637984 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.637941 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.638182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.638066 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.638241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.638173 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dpm\" (UniqueName: \"kubernetes.io/projected/96dac90e-7f3c-4db1-b930-6c578ff38b59-kube-api-access-j7dpm\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.739641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.739604 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dpm\" (UniqueName: \"kubernetes.io/projected/96dac90e-7f3c-4db1-b930-6c578ff38b59-kube-api-access-j7dpm\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.739857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.739784 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.739857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.739840 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.740498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.740476 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.742387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.742360 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.803079 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.803046 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dpm\" (UniqueName: \"kubernetes.io/projected/96dac90e-7f3c-4db1-b930-6c578ff38b59-kube-api-access-j7dpm\") pod \"maas-api-revoke-sub-45ce6493-w7qrj\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:22.821331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:22.821300 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:23.265611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.258789 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj"] Nov 03 19:02:23.338851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.338820 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:02:23.338964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.338910 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:02:23.551224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.551123 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" event={"ID":"4eec5248-4944-46a7-bff9-5da030569f0f","Type":"ContainerStarted","Data":"10dee66c90bcd35242de21739908f926ce49594d392bfd6dbd4542625b9efa3c"} Nov 03 19:02:23.551600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.551220 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" event={"ID":"4eec5248-4944-46a7-bff9-5da030569f0f","Type":"ContainerStarted","Data":"76ed12db59eb77db990fc2d7e4311ff3bf2c3122512cfdf1f15897d821f59ebe"} Nov 03 19:02:23.551600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.551303 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:23.553438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.553268 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" event={"ID":"96dac90e-7f3c-4db1-b930-6c578ff38b59","Type":"ContainerStarted","Data":"b02e799e83c574b4b4087df75f61430a9eb5423673e2ec9352e1cd5c6011d1d3"} Nov 03 19:02:23.699471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.699408 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" podStartSLOduration=2.699390131 podStartE2EDuration="2.699390131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:23.697666195 +0000 UTC m=+1473.180662110" watchObservedRunningTime="2026-11-03 19:02:23.699390131 +0000 UTC m=+1473.182386046" Nov 03 19:02:23.745662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.745631 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:23.862904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.862813 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4df7583-55ad-4641-96dc-49b643910558-kube-api-access-pprgf\" (UniqueName: \"kubernetes.io/projected/c4df7583-55ad-4641-96dc-49b643910558-kube-api-access-pprgf\") pod \"c4df7583-55ad-4641-96dc-49b643910558\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " Nov 03 19:02:23.862904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.862884 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4df7583-55ad-4641-96dc-49b643910558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4df7583-55ad-4641-96dc-49b643910558-maas-api-openshift-ca\") pod \"c4df7583-55ad-4641-96dc-49b643910558\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " Nov 03 19:02:23.863161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.863140 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4df7583-55ad-4641-96dc-49b643910558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4df7583-55ad-4641-96dc-49b643910558-maas-api-xks-ca\") pod \"c4df7583-55ad-4641-96dc-49b643910558\" (UID: \"c4df7583-55ad-4641-96dc-49b643910558\") " Nov 03 19:02:23.866320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.866287 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4df7583-55ad-4641-96dc-49b643910558-maas-api-openshift-ca" pod "c4df7583-55ad-4641-96dc-49b643910558" (UID: "c4df7583-55ad-4641-96dc-49b643910558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:23.871873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.871843 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4df7583-55ad-4641-96dc-49b643910558-maas-api-xks-ca" pod "c4df7583-55ad-4641-96dc-49b643910558" (UID: "c4df7583-55ad-4641-96dc-49b643910558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:23.871873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.871870 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4df7583-55ad-4641-96dc-49b643910558-kube-api-access-pprgf" pod "c4df7583-55ad-4641-96dc-49b643910558" (UID: "c4df7583-55ad-4641-96dc-49b643910558"). InnerVolumeSpecName "kube-api-access-pprgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:23.970271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.970182 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4df7583-55ad-4641-96dc-49b643910558-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:23.970271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.970224 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pprgf\" (UniqueName: \"kubernetes.io/projected/c4df7583-55ad-4641-96dc-49b643910558-kube-api-access-pprgf\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:23.970271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:23.970240 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4df7583-55ad-4641-96dc-49b643910558-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:24.286966 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.286813 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c"] Nov 03 19:02:24.287857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.287831 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c4df7583-55ad-4641-96dc-49b643910558" containerName="revoke-keys" Nov 03 19:02:24.287857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.287857 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4df7583-55ad-4641-96dc-49b643910558" containerName="revoke-keys" Nov 03 19:02:24.288096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.288080 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4df7583-55ad-4641-96dc-49b643910558" containerName="revoke-keys" Nov 03 19:02:24.292029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.291918 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:24.331033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.331005 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c"] Nov 03 19:02:24.347019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.346959 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-30b92815\"" Nov 03 19:02:24.375997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.375964 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh"] Nov 03 19:02:24.381209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.381184 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.442110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.439683 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-30b92815-dockercfg-dllt4\"" Nov 03 19:02:24.485759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.478184 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhd2\" (UniqueName: \"kubernetes.io/projected/2a226ada-8eab-4c72-b8f4-26e476e9dece-kube-api-access-9qhd2\") pod \"payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:24.568975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.558890 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a226ada-8eab-4c72-b8f4-26e476e9dece-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:24.645821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.645780 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh"] Nov 03 19:02:24.662714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.662660 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de5c44-10cd-46c3-8386-25ad4b135c5d-plugins-config-volume\") pod \"payload-processing-e2e-derive-30b92815-7984849f75-2fvrh\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.663025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.663003 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhd2\" (UniqueName: \"kubernetes.io/projected/2a226ada-8eab-4c72-b8f4-26e476e9dece-kube-api-access-9qhd2\") pod \"payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:24.663190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.663178 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a226ada-8eab-4c72-b8f4-26e476e9dece-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:24.663396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.663385 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qd98\" (UniqueName: \"kubernetes.io/projected/b2de5c44-10cd-46c3-8386-25ad4b135c5d-kube-api-access-7qd98\") pod \"payload-processing-e2e-derive-30b92815-7984849f75-2fvrh\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.664828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.664772 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a226ada-8eab-4c72-b8f4-26e476e9dece-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:24.704766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.704598 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" event={"ID":"c4df7583-55ad-4641-96dc-49b643910558","Type":"ContainerDied","Data":"1a1b6d5e7aa19e76d4946c984d2898e603735ef9782bec0e31619cabc7deff4e"} Nov 03 19:02:24.705023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.705005 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a1b6d5e7aa19e76d4946c984d2898e603735ef9782bec0e31619cabc7deff4e" Nov 03 19:02:24.705213 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.705200 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7" Nov 03 19:02:24.709004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.708975 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" event={"ID":"96dac90e-7f3c-4db1-b930-6c578ff38b59","Type":"ContainerStarted","Data":"09ba816d6f9efa711d9a3b23cc2933dbc13e1491b1612b0d5e155403aeeb99bb"} Nov 03 19:02:24.727060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.727030 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhd2\" (UniqueName: \"kubernetes.io/projected/2a226ada-8eab-4c72-b8f4-26e476e9dece-kube-api-access-9qhd2\") pod \"payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:24.764512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.764479 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qd98\" (UniqueName: \"kubernetes.io/projected/b2de5c44-10cd-46c3-8386-25ad4b135c5d-kube-api-access-7qd98\") pod \"payload-processing-e2e-derive-30b92815-7984849f75-2fvrh\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.764671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.764529 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de5c44-10cd-46c3-8386-25ad4b135c5d-plugins-config-volume\") pod \"payload-processing-e2e-derive-30b92815-7984849f75-2fvrh\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.765400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.765374 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de5c44-10cd-46c3-8386-25ad4b135c5d-plugins-config-volume\") pod \"payload-processing-e2e-derive-30b92815-7984849f75-2fvrh\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.827480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.827401 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qd98\" (UniqueName: \"kubernetes.io/projected/b2de5c44-10cd-46c3-8386-25ad4b135c5d-kube-api-access-7qd98\") pod \"payload-processing-e2e-derive-30b92815-7984849f75-2fvrh\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.840349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.840316 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:24.919367 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:24.919337 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:25.226871 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:25.226827 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2de5c44_10cd_46c3_8386_25ad4b135c5d.slice/crio-3b4d5419d66aef98fb4ed381b8ff58f53dd3266fc34b33f2858759d0b1559df6 WatchSource:0}: Error finding container 3b4d5419d66aef98fb4ed381b8ff58f53dd3266fc34b33f2858759d0b1559df6: Status 404 returned error can't find the container with id 3b4d5419d66aef98fb4ed381b8ff58f53dd3266fc34b33f2858759d0b1559df6 Nov 03 19:02:25.229347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.229323 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh"] Nov 03 19:02:25.339519 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:25.339492 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a226ada_8eab_4c72_b8f4_26e476e9dece.slice/crio-2bf7fcf27ed358f8765d39e5fe730611f653845ed2a979d832e2c5bc1ec99f42 WatchSource:0}: Error finding container 2bf7fcf27ed358f8765d39e5fe730611f653845ed2a979d832e2c5bc1ec99f42: Status 404 returned error can't find the container with id 2bf7fcf27ed358f8765d39e5fe730611f653845ed2a979d832e2c5bc1ec99f42 Nov 03 19:02:25.341979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.341955 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c"] Nov 03 19:02:25.720757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.720719 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" event={"ID":"b2de5c44-10cd-46c3-8386-25ad4b135c5d","Type":"ContainerStarted","Data":"ac68b77d12a7c8f8c05ee7bdf48309437c2ce9409029943ceb14c1a96e116c85"} Nov 03 19:02:25.721253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.720778 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" event={"ID":"b2de5c44-10cd-46c3-8386-25ad4b135c5d","Type":"ContainerStarted","Data":"3b4d5419d66aef98fb4ed381b8ff58f53dd3266fc34b33f2858759d0b1559df6"} Nov 03 19:02:25.721253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.720807 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:25.722614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.722590 4948 generic.go:356] "Generic (PLEG): container finished" podID="96dac90e-7f3c-4db1-b930-6c578ff38b59" containerID="09ba816d6f9efa711d9a3b23cc2933dbc13e1491b1612b0d5e155403aeeb99bb" exitCode=0 Nov 03 19:02:25.722762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.722669 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" event={"ID":"96dac90e-7f3c-4db1-b930-6c578ff38b59","Type":"ContainerDied","Data":"09ba816d6f9efa711d9a3b23cc2933dbc13e1491b1612b0d5e155403aeeb99bb"} Nov 03 19:02:25.725187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.725152 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" event={"ID":"2a226ada-8eab-4c72-b8f4-26e476e9dece","Type":"ContainerStarted","Data":"3f7f5995a87da8fd231741f1927a9cf884673cae09a00bda707c92e1af792145"} Nov 03 19:02:25.725313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.725221 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" event={"ID":"2a226ada-8eab-4c72-b8f4-26e476e9dece","Type":"ContainerStarted","Data":"2bf7fcf27ed358f8765d39e5fe730611f653845ed2a979d832e2c5bc1ec99f42"} Nov 03 19:02:25.725313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.725259 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:25.840689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.840617 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" podStartSLOduration=1.840600818 podStartE2EDuration="1.840600818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:25.839652863 +0000 UTC m=+1475.322648775" watchObservedRunningTime="2026-11-03 19:02:25.840600818 +0000 UTC m=+1475.323596738" Nov 03 19:02:25.887145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.887119 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:26.002081 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.997282 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96dac90e-7f3c-4db1-b930-6c578ff38b59-kube-api-access-j7dpm\" (UniqueName: \"kubernetes.io/projected/96dac90e-7f3c-4db1-b930-6c578ff38b59-kube-api-access-j7dpm\") pod \"96dac90e-7f3c-4db1-b930-6c578ff38b59\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " Nov 03 19:02:26.002081 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:25.997503 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-openshift-ca\") pod \"96dac90e-7f3c-4db1-b930-6c578ff38b59\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " Nov 03 19:02:26.005893 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.005848 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-xks-ca\") pod \"96dac90e-7f3c-4db1-b930-6c578ff38b59\" (UID: \"96dac90e-7f3c-4db1-b930-6c578ff38b59\") " Nov 03 19:02:26.008991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.008949 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-openshift-ca" pod "96dac90e-7f3c-4db1-b930-6c578ff38b59" (UID: "96dac90e-7f3c-4db1-b930-6c578ff38b59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:26.013602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.013513 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" podStartSLOduration=2.013492481 podStartE2EDuration="2.013492481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:25.993215321 +0000 UTC m=+1475.476211239" watchObservedRunningTime="2026-11-03 19:02:26.013492481 +0000 UTC m=+1475.496488398" Nov 03 19:02:26.019265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.019233 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-xks-ca" pod "96dac90e-7f3c-4db1-b930-6c578ff38b59" (UID: "96dac90e-7f3c-4db1-b930-6c578ff38b59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:26.035296 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.033814 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dac90e-7f3c-4db1-b930-6c578ff38b59-kube-api-access-j7dpm" pod "96dac90e-7f3c-4db1-b930-6c578ff38b59" (UID: "96dac90e-7f3c-4db1-b930-6c578ff38b59"). InnerVolumeSpecName "kube-api-access-j7dpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:26.072684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.072634 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5"] Nov 03 19:02:26.090295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.090008 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aee09a-t4lz5"] Nov 03 19:02:26.144085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.143532 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:26.144085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.143569 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dpm\" (UniqueName: \"kubernetes.io/projected/96dac90e-7f3c-4db1-b930-6c578ff38b59-kube-api-access-j7dpm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:26.144085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.143585 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96dac90e-7f3c-4db1-b930-6c578ff38b59-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:26.737818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.737789 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" Nov 03 19:02:26.737818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.737809 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj" event={"ID":"96dac90e-7f3c-4db1-b930-6c578ff38b59","Type":"ContainerDied","Data":"b02e799e83c574b4b4087df75f61430a9eb5423673e2ec9352e1cd5c6011d1d3"} Nov 03 19:02:26.738363 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.737887 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02e799e83c574b4b4087df75f61430a9eb5423673e2ec9352e1cd5c6011d1d3" Nov 03 19:02:26.757245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:26.757214 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56b4d7b-f600-4058-8ab8-7ede8bf2357a" path="/var/lib/kubelet/pods/c56b4d7b-f600-4058-8ab8-7ede8bf2357a/volumes" Nov 03 19:02:32.795336 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:02:32.795245 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:02:33.228925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.228890 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75"] Nov 03 19:02:33.229589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.229576 4948 cpu_manager.go:414] "containerMap: removing container" podUID="96dac90e-7f3c-4db1-b930-6c578ff38b59" containerName="revoke-keys" Nov 03 19:02:33.229632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.229590 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96dac90e-7f3c-4db1-b930-6c578ff38b59" containerName="revoke-keys" Nov 03 19:02:33.229733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.229722 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="96dac90e-7f3c-4db1-b930-6c578ff38b59" containerName="revoke-keys" Nov 03 19:02:33.235244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.235219 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.291756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.291722 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75"] Nov 03 19:02:33.334053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.334026 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:02:33.334256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.334077 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:02:33.334256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.334086 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:02:33.339763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.339737 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:02:33.361309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.361282 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.361480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.361462 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.361552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.361507 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjklx\" (UniqueName: \"kubernetes.io/projected/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-kube-api-access-pjklx\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.463273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.463235 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.463484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.463356 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.463484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.463398 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjklx\" (UniqueName: \"kubernetes.io/projected/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-kube-api-access-pjklx\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.464209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.464182 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.465646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.465619 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.525504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.525430 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjklx\" (UniqueName: \"kubernetes.io/projected/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-kube-api-access-pjklx\") pod \"maas-api-revoke-sub-7397a1cf-fwh75\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.551376 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.551346 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:33.947129 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:33.947094 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48157e9e_c9b0_4372_8167_9df5b0c6b9c7.slice/crio-47da65f43abdc5c1c8110e493b3898c1cb41658176951bc9db6f4fbf236a39d2 WatchSource:0}: Error finding container 47da65f43abdc5c1c8110e493b3898c1cb41658176951bc9db6f4fbf236a39d2: Status 404 returned error can't find the container with id 47da65f43abdc5c1c8110e493b3898c1cb41658176951bc9db6f4fbf236a39d2 Nov 03 19:02:33.951753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:33.950459 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75"] Nov 03 19:02:34.082390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.082357 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw"] Nov 03 19:02:34.091236 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.091210 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.142727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.142680 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw"] Nov 03 19:02:34.175637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.175588 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176016 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176058 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176140 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176205 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dhg\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-kube-api-access-j2dhg\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176287 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176392 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176449 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176430 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.176489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.176458 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56de3f-b47d-4824-9560-96dc8e957dff-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.270117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.270032 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz"] Nov 03 19:02:34.275550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.275525 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.277653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.277621 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.277811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.277720 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dhg\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-kube-api-access-j2dhg\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.277879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.277811 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.277985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.277963 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278011 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278041 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56de3f-b47d-4824-9560-96dc8e957dff-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278161 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278208 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278233 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278270 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278641 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278752 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.278920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.278899 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.279109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.279087 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56de3f-b47d-4824-9560-96dc8e957dff-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.280446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.280419 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.281585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.281560 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.301124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.301093 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx"] Nov 03 19:02:34.305871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.305854 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.330639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.330604 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz"] Nov 03 19:02:34.340235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.339954 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.340235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.340090 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dhg\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-kube-api-access-j2dhg\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.380118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380056 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.380118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380098 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380142 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380174 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380203 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380251 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380294 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380353 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380409 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380484 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380518 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rjv\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-kube-api-access-j9rjv\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380636 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380664 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380735 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380771 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f50c444-5018-444d-b775-d40e3d4bf87a-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380803 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxm7\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-kube-api-access-mgxm7\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380838 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.381203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.380874 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.398823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.398785 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx"] Nov 03 19:02:34.407231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.407194 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:34.483862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.483818 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.485666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.485641 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.485818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.484162 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.485930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.485913 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.486374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.486355 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.486539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.486522 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.486666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.486649 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rjv\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-kube-api-access-j9rjv\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.486770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.486716 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.486971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.486954 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.487049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.486984 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.487148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.487132 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.488133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.488113 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f50c444-5018-444d-b775-d40e3d4bf87a-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.488295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.488278 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxm7\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-kube-api-access-mgxm7\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.488454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.488437 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.488619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.488601 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.491828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.489158 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.491828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.489416 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.491828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.488672 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f50c444-5018-444d-b775-d40e3d4bf87a-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.491828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.489716 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.492652 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.492637 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.494505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.494474 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.495356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.495335 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.495758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.495745 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.496161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.496144 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.497242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.495262 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.497464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.496080 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.497610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.493107 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.497762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.495684 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.498751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.498676 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.501471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.501447 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.506193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.506137 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.509423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.509400 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.561950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.561861 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.564185 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.564118 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.564531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.564485 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rjv\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-kube-api-access-j9rjv\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.566168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.566147 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxm7\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-kube-api-access-mgxm7\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.593523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.593219 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:34.620535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.620504 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:34.722084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.722056 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:02:34.820642 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:34.820601 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee56de3f_b47d_4824_9560_96dc8e957dff.slice/crio-12a71c065065aa057427997ff907c6e38a47e1ccd4c317a9749cf1d7552afa0e WatchSource:0}: Error finding container 12a71c065065aa057427997ff907c6e38a47e1ccd4c317a9749cf1d7552afa0e: Status 404 returned error can't find the container with id 12a71c065065aa057427997ff907c6e38a47e1ccd4c317a9749cf1d7552afa0e Nov 03 19:02:34.823980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.823936 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw"] Nov 03 19:02:34.825729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.825657 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:34.825834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.825753 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:34.825834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.825795 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:34.832510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.832482 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" event={"ID":"48157e9e-c9b0-4372-8167-9df5b0c6b9c7","Type":"ContainerStarted","Data":"6c4c3cd349aaef577a77d19f8ae9f7a379f9d7c15b50a2e6ab695ab6e811cd1e"} Nov 03 19:02:34.832618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.832533 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" event={"ID":"48157e9e-c9b0-4372-8167-9df5b0c6b9c7","Type":"ContainerStarted","Data":"47da65f43abdc5c1c8110e493b3898c1cb41658176951bc9db6f4fbf236a39d2"} Nov 03 19:02:34.986059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:34.985885 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" podStartSLOduration=1.9858634579999999 podStartE2EDuration="1.985863458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:34.979227541 +0000 UTC m=+1484.462223456" watchObservedRunningTime="2026-11-03 19:02:34.985863458 +0000 UTC m=+1484.468859373" Nov 03 19:02:35.006742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.006663 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz"] Nov 03 19:02:35.010020 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:35.009989 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98895b9_c0ff_4d5a_b615_ba0a0227f1fc.slice/crio-2299a7aa5a55c3cf96f3b9bd46d065bb9777ed3163b64a17c759305503060758 WatchSource:0}: Error finding container 2299a7aa5a55c3cf96f3b9bd46d065bb9777ed3163b64a17c759305503060758: Status 404 returned error can't find the container with id 2299a7aa5a55c3cf96f3b9bd46d065bb9777ed3163b64a17c759305503060758 Nov 03 19:02:35.016792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.016750 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:35.016888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.016846 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:35.016930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.016893 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:35.059052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.059025 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx"] Nov 03 19:02:35.061254 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:35.061225 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f50c444_5018_444d_b775_d40e3d4bf87a.slice/crio-afcff45ed272d601807193f492463aad066b9b3792beb919f2af9359c18546db WatchSource:0}: Error finding container afcff45ed272d601807193f492463aad066b9b3792beb919f2af9359c18546db: Status 404 returned error can't find the container with id afcff45ed272d601807193f492463aad066b9b3792beb919f2af9359c18546db Nov 03 19:02:35.065806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.065770 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:35.065897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.065842 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:35.065897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.065877 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:35.107500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.107475 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d"] Nov 03 19:02:35.113002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.112978 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.205222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.205192 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d"] Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.220710 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.220879 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.220952 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.221024 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.221105 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdgc\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-kube-api-access-vhdgc\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.221147 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.221204 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.221261 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.221840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.221292 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323025 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323116 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdgc\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-kube-api-access-vhdgc\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323156 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323205 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323252 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323279 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323336 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323461 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.323520 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.324776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.324265 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.344902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.341143 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.344902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.341868 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.344902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.342117 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.344902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.342813 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.344902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.343834 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.344902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.344733 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.387719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.387669 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.388909 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.388856 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdgc\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-kube-api-access-vhdgc\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.500151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.500104 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:35.668936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.668895 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j"] Nov 03 19:02:35.732068 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.732033 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.741196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.741162 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j"] Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838184 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838277 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838385 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838473 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838545 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838676 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838727 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838768 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.845817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.838804 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4bl\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-kube-api-access-nq4bl\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.892683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.892162 4948 generic.go:356] "Generic (PLEG): container finished" podID="48157e9e-c9b0-4372-8167-9df5b0c6b9c7" containerID="6c4c3cd349aaef577a77d19f8ae9f7a379f9d7c15b50a2e6ab695ab6e811cd1e" exitCode=0 Nov 03 19:02:35.892683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.892254 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" event={"ID":"48157e9e-c9b0-4372-8167-9df5b0c6b9c7","Type":"ContainerDied","Data":"6c4c3cd349aaef577a77d19f8ae9f7a379f9d7c15b50a2e6ab695ab6e811cd1e"} Nov 03 19:02:35.914751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.913165 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" event={"ID":"ee56de3f-b47d-4824-9560-96dc8e957dff","Type":"ContainerStarted","Data":"b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd"} Nov 03 19:02:35.914751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.913235 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" event={"ID":"ee56de3f-b47d-4824-9560-96dc8e957dff","Type":"ContainerStarted","Data":"12a71c065065aa057427997ff907c6e38a47e1ccd4c317a9749cf1d7552afa0e"} Nov 03 19:02:35.918555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.918460 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" event={"ID":"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc","Type":"ContainerStarted","Data":"70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b"} Nov 03 19:02:35.918555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.918517 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" event={"ID":"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc","Type":"ContainerStarted","Data":"2299a7aa5a55c3cf96f3b9bd46d065bb9777ed3163b64a17c759305503060758"} Nov 03 19:02:35.923168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.923085 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" event={"ID":"6f50c444-5018-444d-b775-d40e3d4bf87a","Type":"ContainerStarted","Data":"f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144"} Nov 03 19:02:35.923168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.923137 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" event={"ID":"6f50c444-5018-444d-b775-d40e3d4bf87a","Type":"ContainerStarted","Data":"afcff45ed272d601807193f492463aad066b9b3792beb919f2af9359c18546db"} Nov 03 19:02:35.941485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.941113 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.942161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.942112 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.942669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.942553 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.947613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.943385 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.947613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.942630 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.947613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.944320 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4bl\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-kube-api-access-nq4bl\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.947613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.944691 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.947613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.944811 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.947613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.945280 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.950660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.945477 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:35.978061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.978026 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.002466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.947887 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.003010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.949566 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.003128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.947564 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.003224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:35.979220 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.003306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.001124 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.014601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.014467 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d"] Nov 03 19:02:36.034685 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:36.034649 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7138e7af_3d7a_4c4e_9ce5_e26f2a024d0c.slice/crio-8a1badbcf9fe072d85f37ffcc70b6b9c6533a93efd33847d1b242ef20a8af039 WatchSource:0}: Error finding container 8a1badbcf9fe072d85f37ffcc70b6b9c6533a93efd33847d1b242ef20a8af039: Status 404 returned error can't find the container with id 8a1badbcf9fe072d85f37ffcc70b6b9c6533a93efd33847d1b242ef20a8af039 Nov 03 19:02:36.036801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.036768 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:36.036968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.036952 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:36.037061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.037049 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:36.136146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.136066 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.136635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.136610 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4bl\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-kube-api-access-nq4bl\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.351721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.331161 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podStartSLOduration=2.331135366 podStartE2EDuration="2.331135366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:36.229251132 +0000 UTC m=+1485.712247047" watchObservedRunningTime="2026-11-03 19:02:36.331135366 +0000 UTC m=+1485.814131281" Nov 03 19:02:36.366723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.357910 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:36.400747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.395378 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podStartSLOduration=2.395357469 podStartE2EDuration="2.395357469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:36.379924621 +0000 UTC m=+1485.862920537" watchObservedRunningTime="2026-11-03 19:02:36.395357469 +0000 UTC m=+1485.878353383" Nov 03 19:02:36.412845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.412788 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:36.465738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.420885 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:36.465738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.420947 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:36.605792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.599539 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:36.605792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.600903 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podStartSLOduration=2.600885278 podStartE2EDuration="2.600885278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:36.549611507 +0000 UTC m=+1486.032607423" watchObservedRunningTime="2026-11-03 19:02:36.600885278 +0000 UTC m=+1486.083881193" Nov 03 19:02:36.620726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.612848 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:36.620726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.612909 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:36.620988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.620863 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:36.622924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.622884 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:36.629873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:36.629832 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:37.277977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.264444 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:02:37.277977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.264627 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:02:37.588398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.467931 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.588398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.467994 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:37.602279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.602128 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.602279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.602192 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:37.659779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.630110 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.659779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.630177 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:37.770394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.763597 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:37.770394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.763690 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:37.770394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.763762 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:02:37.906091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.888983 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:37.906091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:37.890682 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:38.015045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.014986 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:38.079738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.059648 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" event={"ID":"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c","Type":"ContainerStarted","Data":"2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39"} Nov 03 19:02:38.079738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.059736 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" event={"ID":"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c","Type":"ContainerStarted","Data":"8a1badbcf9fe072d85f37ffcc70b6b9c6533a93efd33847d1b242ef20a8af039"} Nov 03 19:02:38.103629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.102066 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:38.126292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.120012 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Nov 03 19:02:38.301735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.296859 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j"] Nov 03 19:02:38.518725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.494186 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.518725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.494259 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:38.571917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.571820 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:38.572547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.572509 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.572816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.572777 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:38.603839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.603794 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.604150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.604103 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:38.622738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.622569 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.622738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.622640 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:38.946815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.911625 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:02:38.950609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:38.916727 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:39.112771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.107794 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.145356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.145321 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq"] Nov 03 19:02:39.298763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.294130 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq"] Nov 03 19:02:39.298763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.294234 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:02:39.298763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.294277 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:02:39.298763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.294311 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:02:39.298763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.294779 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:39.607641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.510222 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podStartSLOduration=4.510201361 podStartE2EDuration="4.510201361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:38.209664002 +0000 UTC m=+1487.692659920" watchObservedRunningTime="2026-11-03 19:02:39.510201361 +0000 UTC m=+1488.993197277" Nov 03 19:02:39.624906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.568084 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.625209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.625153 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:39.625368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.608805 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.705737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.667295 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.714738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.707026 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:39.721824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.586923 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.771772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.570463 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.772147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.772030 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:39.772349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.743083 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.803030 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.802822 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Nov 03 19:02:39.867862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.736460 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.868239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.868200 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:39.957939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.957889 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.963981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:39.963920 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.120800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.073004 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.120800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.073093 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmxv\" (UniqueName: \"kubernetes.io/projected/aca36255-a69e-4c97-88a3-7af6fb78c099-kube-api-access-cqmxv\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.120800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.073248 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.163286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.154603 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.188229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.188182 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:40.191137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.191110 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.191361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.191323 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmxv\" (UniqueName: \"kubernetes.io/projected/aca36255-a69e-4c97-88a3-7af6fb78c099-kube-api-access-cqmxv\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.214013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.213914 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.250220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.235848 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" event={"ID":"8e4b1269-cfed-484b-8957-d8eb9b3dbb43","Type":"ContainerStarted","Data":"d218690925b3c5de637904d15f009f329369a95eec16fb2e8d3c84a60213e8a8"} Nov 03 19:02:40.281681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.281031 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.296726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.282750 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.296726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.283836 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.449131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.433021 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmxv\" (UniqueName: \"kubernetes.io/projected/aca36255-a69e-4c97-88a3-7af6fb78c099-kube-api-access-cqmxv\") pod \"maas-api-revoke-sub-04333afe-7j4jq\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.500962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.499797 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.500962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.499863 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:40.502473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.502363 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.502473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.502413 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:40.508730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.507886 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:40.519726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.509044 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:40.681800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.670825 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.681800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.680016 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.681800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.680068 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:40.681800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.680154 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:02:40.710418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.710385 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-xks-ca\") pod \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " Nov 03 19:02:40.710777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.710753 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-openshift-ca\") pod \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " Nov 03 19:02:40.736624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.710994 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-kube-api-access-pjklx\" (UniqueName: \"kubernetes.io/projected/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-kube-api-access-pjklx\") pod \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\" (UID: \"48157e9e-c9b0-4372-8167-9df5b0c6b9c7\") " Nov 03 19:02:40.753770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.737785 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.753770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.730833 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.753770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.738439 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:40.753770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.738357 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-openshift-ca" pod "48157e9e-c9b0-4372-8167-9df5b0c6b9c7" (UID: "48157e9e-c9b0-4372-8167-9df5b0c6b9c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:40.753770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.738559 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:02:40.780516 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.780311 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:40.860870 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.847925 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:41.076464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.925829 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-xks-ca" pod "48157e9e-c9b0-4372-8167-9df5b0c6b9c7" (UID: "48157e9e-c9b0-4372-8167-9df5b0c6b9c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:41.076464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.989393 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:41.076464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.990589 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:41.076464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:40.991849 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:02:41.076464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.008480 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-kube-api-access-pjklx" pod "48157e9e-c9b0-4372-8167-9df5b0c6b9c7" (UID: "48157e9e-c9b0-4372-8167-9df5b0c6b9c7"). InnerVolumeSpecName "kube-api-access-pjklx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:41.118981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.102926 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjklx\" (UniqueName: \"kubernetes.io/projected/48157e9e-c9b0-4372-8167-9df5b0c6b9c7-kube-api-access-pjklx\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:41.180024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.179977 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.208770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.196159 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.301963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.301915 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.481726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.481269 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:02:41.481726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.481340 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:02:41.481726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.481378 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:02:41.534769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.510422 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:41.564845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.564803 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:41.683732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.630562 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:41.683732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.630622 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:41.683732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.630736 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:41.683732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.630761 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:41.683732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.632143 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:41.683732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.632184 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:41.683732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.677766 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.762721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.759563 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.818140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.787301 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:41.934771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.927814 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 19:02:41.956819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.945196 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" event={"ID":"48157e9e-c9b0-4372-8167-9df5b0c6b9c7","Type":"ContainerDied","Data":"47da65f43abdc5c1c8110e493b3898c1cb41658176951bc9db6f4fbf236a39d2"} Nov 03 19:02:41.957105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.957073 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47da65f43abdc5c1c8110e493b3898c1cb41658176951bc9db6f4fbf236a39d2" Nov 03 19:02:41.967189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:41.957509 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75" Nov 03 19:02:42.018028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.017983 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:42.031066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.031024 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:02:42.128092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.128053 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:42.365725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.362810 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:42.365725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.235002 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:42.365725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.045207 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:42.365725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.300456 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Nov 03 19:02:42.365725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.300907 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" event={"ID":"8e4b1269-cfed-484b-8957-d8eb9b3dbb43","Type":"ContainerStarted","Data":"86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a"} Nov 03 19:02:42.365725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.348657 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:42.379915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.379820 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:42.532736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.530925 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.532736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.531005 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:42.532736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.531936 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.532736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.531991 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:42.532736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.532174 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.532736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.532219 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:42.675424 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.675390 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.949950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.942457 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.038136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.906149 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:02:43.039016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.989909 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:43.101237 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.101077 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:43.127886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.127837 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:43.241871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:42.975639 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.416753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.378114 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.629325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.629209 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 03 19:02:43.630625 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.630176 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:43.631744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.631713 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:43.682437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.682367 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Nov 03 19:02:43.898380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.890658 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:43.921624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.921563 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:43.922351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.922312 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:43.922694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:43.922659 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:44.087407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.087368 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.088125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.088059 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:44.342730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.327487 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.342730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.330040 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:44.342730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.330138 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.387724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.356265 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" podUID="621ec304-b99e-4a18-b966-dd7e1e0c73b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.476781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.368505 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:02:44.476781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.452190 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:02:44.476781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.392650 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.476781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.452894 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:44.476781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.369453 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Nov 03 19:02:44.476781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.470460 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.476781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.470521 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:44.514724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.489815 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.535025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.534983 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:44.566516 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.566471 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:02:44.583725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.579868 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.587879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.587803 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:44.681875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.635005 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.701678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.689827 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.763674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.678142 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" podUID="4b51d6f9-dbec-4fa0-a0b3-072746a6485d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.148:50052\" within 1s: context deadline exceeded" Nov 03 19:02:44.763674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.638761 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:44.763674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.681751 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.763674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.757963 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:44.763674 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:02:44.730081 4948 logging.go:55] [core] [Channel #2531 SubChannel #2532]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.148:50052", ServerName: "10.132.0.148:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.148:50052: operation was canceled" Nov 03 19:02:44.770880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.770839 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:44.774719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.771091 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.929724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.927656 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podStartSLOduration=9.927632844 podStartE2EDuration="9.927632844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:43.486420584 +0000 UTC m=+1492.969416500" watchObservedRunningTime="2026-11-03 19:02:44.927632844 +0000 UTC m=+1494.410628760" Nov 03 19:02:44.944855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.944813 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.972987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.972938 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:44.983730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.982593 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.983730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:44.982859 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:45.173993 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:02:45.158196 4948 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.954s" Nov 03 19:02:45.173993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.158280 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:02:45.300862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.300508 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.363940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.347676 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.371305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.371268 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.378192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.377343 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:45.488901 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.488847 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.666868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.535256 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:45.670826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.670771 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.671048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.651896 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.696124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.612788 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.696415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.696374 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:45.696538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.612836 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.696643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.696626 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:45.696762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.635068 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.696877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.696859 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:45.696973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.654039 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.697067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.697051 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:45.697180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.535427 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.714723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.698800 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.857726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.775780 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.857726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.791543 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Nov 03 19:02:45.976729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:45.967753 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq"] Nov 03 19:02:46.319829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.312457 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:46.321607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.321571 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:02:46.403779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.371078 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:46.403779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.372120 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.403779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.372167 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:46.423763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.409015 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.423763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.409077 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:46.528726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.503401 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.528726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.503466 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:46.595442 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.595346 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.595722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.595670 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:46.719736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.684326 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:02:46.719736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.697897 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:46.719736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.708059 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:02:46.851245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.788593 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.851245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.788651 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:46.851245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.789052 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:46.851245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.803779 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Nov 03 19:02:46.851245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:46.805457 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:47.310809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.289105 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" event={"ID":"aca36255-a69e-4c97-88a3-7af6fb78c099","Type":"ContainerStarted","Data":"4f83f7956d5dc5e3ebd11dabcc3e38654090bda4f75e43698ed27e4c28946a35"} Nov 03 19:02:47.375737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.327864 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:47.375737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.327945 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:02:47.376304 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.375943 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:47.414724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.407642 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:47.414724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.407716 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:47.418893 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.418849 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:47.419177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.419113 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:47.508018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.507955 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:47.508473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.508307 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:47.596772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.596652 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:47.597007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.596980 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:47.625538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.625508 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:02:47.655548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.655514 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:47.655796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.655769 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:47.723727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.718835 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:47.724011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.723975 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:02:47.725185 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.725153 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:47.821822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.792898 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:47.823575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.823524 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:47.993742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:47.964854 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:48.433767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.382878 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" event={"ID":"aca36255-a69e-4c97-88a3-7af6fb78c099","Type":"ContainerStarted","Data":"33b9d66dc15e17d46250eced26b63ce7f20feb87d9380c5795c405a411a5e23f"} Nov 03 19:02:48.433767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.383270 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:48.433767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.383320 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:48.433767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.383388 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:02:48.433767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.384167 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:48.433767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.413799 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:48.433767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.413857 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:48.632057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.631913 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" podStartSLOduration=11.631890205 podStartE2EDuration="11.631890205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:48.488667263 +0000 UTC m=+1497.971663179" watchObservedRunningTime="2026-11-03 19:02:48.631890205 +0000 UTC m=+1498.114886128" Nov 03 19:02:48.688662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.688628 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:48.688944 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.688914 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:48.689136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.689119 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:48.689248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.689231 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:48.708650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.708261 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:48.708650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.708317 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:48.719866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.719809 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:48.724948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.724909 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:48.725972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.725938 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:48.793819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.793771 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:48.838199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:48.837789 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:49.398823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.378288 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:49.398823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.378352 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:49.398823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.385590 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:49.398823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.385999 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:02:49.429754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.412119 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:02:49.429754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.412179 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:02:49.501725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.486880 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:49.521954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.521917 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:49.522223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.522194 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:49.602436 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.602393 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:49.610934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.610887 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:49.773835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.733178 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:49.773835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.746802 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:02:49.773835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.747260 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:02:49.804724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.775234 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:49.805040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.805000 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:49.833691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.833638 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:49.881724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.851008 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:49.881724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.852888 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:49.944846 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:02:49.928176 4948 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.173s" Nov 03 19:02:49.968818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.968779 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:49.969114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:49.969097 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:50.168783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.149795 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" event={"ID":"aca36255-a69e-4c97-88a3-7af6fb78c099","Type":"ContainerDied","Data":"33b9d66dc15e17d46250eced26b63ce7f20feb87d9380c5795c405a411a5e23f"} Nov 03 19:02:50.168783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.149922 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:50.168783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.143301 4948 generic.go:356] "Generic (PLEG): container finished" podID="aca36255-a69e-4c97-88a3-7af6fb78c099" containerID="33b9d66dc15e17d46250eced26b63ce7f20feb87d9380c5795c405a411a5e23f" exitCode=0 Nov 03 19:02:50.408725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.386869 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:50.408725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.396653 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:50.470722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.463457 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:50.495724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.470979 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:50.496079 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.496041 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:50.497109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.497085 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:02:50.535806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.535774 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:50.562726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.509255 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:50.562997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.562965 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:50.666732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.662973 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 19:02:50.666732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.663045 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 19:02:50.666732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.664081 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:02:50.666732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.664127 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:02:51.081903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.972955 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:51.081903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:50.990445 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:51.142469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.142415 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:51.142786 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.142756 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:02:51.472726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.448980 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Nov 03 19:02:51.472726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.450298 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:51.496725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.474982 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:02:51.558989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.558956 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:51.572312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.572278 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:51.639868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.639814 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:51.752385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.752288 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:51.752385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.752350 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:51.813850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.806137 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:02:51.838723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.838088 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:51.838723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.838152 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:51.872841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:51.807952 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:52.044598 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.044502 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:52.045175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.045143 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:52.082985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.082937 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Nov 03 19:02:52.169679 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.169622 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:52.489744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.482396 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:52.543908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.543878 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:02:52.544200 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.544185 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:02:52.562723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.561912 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:52.564816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.563243 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:52.614722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.611339 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:52.614722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.611399 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:52.614722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.614467 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:52.624514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.624478 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:52.624800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.624770 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:52.625603 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.625577 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:52.845803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.836793 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:52.993721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.950571 4948 scope.go:122] "RemoveContainer" containerID="28098d48ce27dc6827b2cc23816548ae334207ac9dd36170b7b973dee204ae6b" Nov 03 19:02:53.000724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:52.997032 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:53.052959 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.046978 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:53.052959 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.047052 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:53.088859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.088810 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:53.135989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.113111 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:02:53.143229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.143185 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 03 19:02:53.320639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.320593 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:02:53.321033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.321002 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:02:53.899189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.827996 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:53.899189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.828372 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:02:53.899189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.834182 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:53.899189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.834230 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:54.038812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.933837 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Nov 03 19:02:54.038812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.010959 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:54.038812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:53.967804 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:54.174806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.148347 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 19:02:54.174806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.148402 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 19:02:54.174806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.153273 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:54.174806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.153317 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:54.174806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.156710 4948 scope.go:122] "RemoveContainer" containerID="1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8" Nov 03 19:02:54.176402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.176188 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:54.176752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.176591 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:54.521537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.439821 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:54.521537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.478305 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:54.637461 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.637428 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:54.642586 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.642559 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:02:54.671387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.671362 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:02:54.712281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.712251 4948 scope.go:122] "RemoveContainer" containerID="dd311bf6521f27fbf8d4191800e44b99a3d7e6f960b2cf81d9d4d42a7f2b7ec3" Nov 03 19:02:54.941737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:54.921215 4948 scope.go:122] "RemoveContainer" containerID="fe527ef832b50c33ab23d6c15b16d2de5b3c46caba928224d2959859bfecf378" Nov 03 19:02:55.028738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.027864 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf" podUID="621ec304-b99e-4a18-b966-dd7e1e0c73b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 03 19:02:55.079880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.042685 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:55.080865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.080826 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:55.119773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.115744 4948 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-v5gvn container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded" start-of-body= Nov 03 19:02:55.119773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.115816 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5gvn" podUID="8f64f0eb-fed7-496b-92a6-84601b0bb544" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded" Nov 03 19:02:55.123108 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.123080 4948 trace.go:236] Trace[370129257]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-2nqz6" (03-Nov-2026 19:02:53.389) (total time: 1733ms): Nov 03 19:02:55.123108 ip-10-0-128-95 kubenswrapper[4948]: Trace[370129257]: [1.733088101s] [1.733088101s] END Nov 03 19:02:55.133580 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.133548 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 19:02:55.133853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.133825 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 19:02:55.166329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.166266 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:02:55.171027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.170995 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:55.171229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.171208 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:55.180294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.179056 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:55.180294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.179160 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:55.308749 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.308723 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:02:55.308992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.308976 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:02:55.309130 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.309118 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:02:55.309264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.309251 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:02:55.309377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.309366 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:02:55.312606 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:02:55.312480 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8\": container with ID starting with 1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8 not found: ID does not exist" containerID="1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8" Nov 03 19:02:55.312606 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:02:55.312547 4948 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8\": container with ID starting with 1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8 not found: ID does not exist" Nov 03 19:02:55.314478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.314405 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:55.336131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.336103 4948 scope.go:122] "RemoveContainer" containerID="a7101d4c8ab3fb68220c78ad87c35fe0fb72151a247844df924b040e97d29123" Nov 03 19:02:55.505484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.505436 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca36255-a69e-4c97-88a3-7af6fb78c099-kube-api-access-cqmxv\" (UniqueName: \"kubernetes.io/projected/aca36255-a69e-4c97-88a3-7af6fb78c099-kube-api-access-cqmxv\") pod \"aca36255-a69e-4c97-88a3-7af6fb78c099\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " Nov 03 19:02:55.505847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.505809 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-xks-ca\") pod \"aca36255-a69e-4c97-88a3-7af6fb78c099\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " Nov 03 19:02:55.555805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.542537 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-openshift-ca\") pod \"aca36255-a69e-4c97-88a3-7af6fb78c099\" (UID: \"aca36255-a69e-4c97-88a3-7af6fb78c099\") " Nov 03 19:02:55.587308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.568922 4948 scope.go:122] "RemoveContainer" containerID="cd67cdab18f10050f03451e8401fb2fb949915120e2cfadf0d11baac3fae69ba" Nov 03 19:02:55.587674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.571794 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-openshift-ca" pod "aca36255-a69e-4c97-88a3-7af6fb78c099" (UID: "aca36255-a69e-4c97-88a3-7af6fb78c099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:02:55.588085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.587881 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-xks-ca" pod "aca36255-a69e-4c97-88a3-7af6fb78c099" (UID: "aca36255-a69e-4c97-88a3-7af6fb78c099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:02:55.614111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.605031 4948 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8\": container with ID starting with 1ecce1bbfcc9da1a8473619a910de14110055ba56214d2051ec7eff517d984d8 not found: ID does not exist" Nov 03 19:02:55.633757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.633684 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca36255-a69e-4c97-88a3-7af6fb78c099-kube-api-access-cqmxv" pod "aca36255-a69e-4c97-88a3-7af6fb78c099" (UID: "aca36255-a69e-4c97-88a3-7af6fb78c099"). InnerVolumeSpecName "kube-api-access-cqmxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:02:55.645931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.645897 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:55.646111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.646097 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmxv\" (UniqueName: \"kubernetes.io/projected/aca36255-a69e-4c97-88a3-7af6fb78c099-kube-api-access-cqmxv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:55.646212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.646201 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca36255-a69e-4c97-88a3-7af6fb78c099-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:02:55.698722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.697863 4948 scope.go:122] "RemoveContainer" containerID="e3f95ea6a12159834d048fb89372e58e9729032fe4c214019d6bd6700101dacd" Nov 03 19:02:55.720932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.703652 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:55.720932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.703738 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:55.826944 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.824425 4948 scope.go:122] "RemoveContainer" containerID="499077075a67c1111eeaa627b3260358bcb591f8e2b15750c03984461bead9b3" Nov 03 19:02:55.841485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.841393 4948 scope.go:122] "RemoveContainer" containerID="cf0ab181fae2c37b8cbaefb845c8727850ecce81bcc204a42800a89abfcb7da9" Nov 03 19:02:55.848167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.848127 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" event={"ID":"aca36255-a69e-4c97-88a3-7af6fb78c099","Type":"ContainerDied","Data":"4f83f7956d5dc5e3ebd11dabcc3e38654090bda4f75e43698ed27e4c28946a35"} Nov 03 19:02:55.848289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.848182 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f83f7956d5dc5e3ebd11dabcc3e38654090bda4f75e43698ed27e4c28946a35" Nov 03 19:02:55.849193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.849170 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq" Nov 03 19:02:55.908897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.908868 4948 scope.go:122] "RemoveContainer" containerID="42e43a7d9bd2ffd05781496576ae3e99580bf3e54e14d540ec2d0aabfd59fb2c" Nov 03 19:02:55.937235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:55.937108 4948 scope.go:122] "RemoveContainer" containerID="61a62d03786067fd672ead3076b95f9a2d974ed1991ed5c82c6d9956dd181ae7" Nov 03 19:02:56.276691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:56.276656 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:56.279218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:56.279191 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:02:56.745837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:56.745806 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:56.746012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:56.745854 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:57.110778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:57.110673 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:02:57.112204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:02:57.112180 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:03:02.799746 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:03:02.799414 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:03:09.061035 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:09.061001 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:03:19.014725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:18.998787 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:03:19.350799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.349907 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.401727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.397869 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.682869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.682762 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:03:19.796903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.796832 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.826651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.826589 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:03:19.865917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.865857 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:03:19.866255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.866207 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.877778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.877733 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.994377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:19.994244 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:03:20.030842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.024797 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:03:20.030842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.025265 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:03:20.356990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.356866 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.362425 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.362396 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:03:20.372397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.372360 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.473724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.454515 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.608692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.608602 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:03:20.723021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.722961 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.724185 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.724155 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:03:20.879534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.839558 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:03:20.897046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:20.882790 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:21.017747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.008619 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:21.058195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.041270 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 19:03:21.058195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.052284 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": context deadline exceeded" Nov 03 19:03:21.267757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.254354 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:03:21.267757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.254491 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:21.365082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.364878 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:03:21.437734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.382265 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Nov 03 19:03:21.622505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.622400 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:03:21.804724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.759323 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:21.876840 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:03:21.844539 4948 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.094s" Nov 03 19:03:21.931235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.931208 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:03:21.931561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.931542 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:03:21.959787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.915970 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:03:22.032734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.849024 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:22.083920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.915929 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:22.084495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.959434 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:22.084632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:21.960019 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:03:22.084849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.084818 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:03:22.115341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.046293 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:22.188095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.014060 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:03:22.392724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.365806 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:22.415247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.384776 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:22.422245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.419193 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:03:22.463727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.454219 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:03:22.666908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:22.666868 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:23.007955 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.007906 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:23.199925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.199882 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:23.250362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.234777 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:03:23.250362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.235677 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:03:23.250362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.235841 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:03:23.250362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.235886 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:03:23.255222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.255191 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:03:23.261652 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.261322 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:03:23.278953 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:23.278929 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:03:32.795658 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:03:32.795616 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:03:46.719879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.719841 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c"] Nov 03 19:03:46.720649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.720624 4948 cpu_manager.go:414] "containerMap: removing container" podUID="aca36255-a69e-4c97-88a3-7af6fb78c099" containerName="revoke-keys" Nov 03 19:03:46.720874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.720842 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca36255-a69e-4c97-88a3-7af6fb78c099" containerName="revoke-keys" Nov 03 19:03:46.721002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.720894 4948 cpu_manager.go:414] "containerMap: removing container" podUID="48157e9e-c9b0-4372-8167-9df5b0c6b9c7" containerName="revoke-keys" Nov 03 19:03:46.721002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.720903 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48157e9e-c9b0-4372-8167-9df5b0c6b9c7" containerName="revoke-keys" Nov 03 19:03:46.721270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.721248 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca36255-a69e-4c97-88a3-7af6fb78c099" containerName="revoke-keys" Nov 03 19:03:46.721270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.721273 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="48157e9e-c9b0-4372-8167-9df5b0c6b9c7" containerName="revoke-keys" Nov 03 19:03:46.724921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.724905 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.812838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.812797 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.812872 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.812996 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.813056 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d026d-b159-4d44-84ef-a20554d43c80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.813176 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.813236 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.813415 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4dt\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-kube-api-access-jr4dt\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.813462 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.813553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.813540 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.846230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.846187 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c"] Nov 03 19:03:46.915307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915270 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915359 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915396 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d026d-b159-4d44-84ef-a20554d43c80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915431 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915464 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915587 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4dt\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-kube-api-access-jr4dt\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915633 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915714 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915830 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.915952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.915908 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.916109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.916050 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.916257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.916236 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.916343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.916321 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.916474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.916451 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d026d-b159-4d44-84ef-a20554d43c80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.918040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.918020 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.918267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.918250 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.978211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.978138 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:46.978352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:46.978264 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4dt\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-kube-api-access-jr4dt\") pod \"maas-default-gateway-openshift-default-7854949c9-m4w4c\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:47.039568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.039537 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:47.555200 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:03:47.555166 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3d026d_b159_4d44_84ef_a20554d43c80.slice/crio-ddfccda635e28d1b384e1489a8c64e2e5dd705d52af857dafbfad945a39eace0 WatchSource:0}: Error finding container ddfccda635e28d1b384e1489a8c64e2e5dd705d52af857dafbfad945a39eace0: Status 404 returned error can't find the container with id ddfccda635e28d1b384e1489a8c64e2e5dd705d52af857dafbfad945a39eace0 Nov 03 19:03:47.557511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.557484 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c"] Nov 03 19:03:47.558043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.558009 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:47.558143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.558077 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:47.558143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.558110 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:47.865248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.865145 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" event={"ID":"9a3d026d-b159-4d44-84ef-a20554d43c80","Type":"ContainerStarted","Data":"ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90"} Nov 03 19:03:47.865248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.865223 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" event={"ID":"9a3d026d-b159-4d44-84ef-a20554d43c80","Type":"ContainerStarted","Data":"ddfccda635e28d1b384e1489a8c64e2e5dd705d52af857dafbfad945a39eace0"} Nov 03 19:03:47.989751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:47.989670 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podStartSLOduration=1.989652491 podStartE2EDuration="1.989652491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:47.988545509 +0000 UTC m=+1557.471541424" watchObservedRunningTime="2026-11-03 19:03:47.989652491 +0000 UTC m=+1557.472648406" Nov 03 19:03:48.040599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:48.040569 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:48.042264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:48.042232 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:48.042432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:48.042302 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:49.040455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.040419 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:49.040865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.040475 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:49.711443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.711408 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt"] Nov 03 19:03:49.716765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.716738 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.781934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.781885 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.782268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.782236 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.782437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.782419 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.782668 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.782646 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.782857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.782835 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949r8\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-kube-api-access-949r8\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.782992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.782975 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.783173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.783156 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29a321b-415d-413f-9d6e-d38150ef8e67-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.783324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.783303 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.783456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.783439 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.790940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.790910 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz"] Nov 03 19:03:49.796075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.796057 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.819977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.819950 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt"] Nov 03 19:03:49.885721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.885051 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.885721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.885144 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.885721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.885184 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.885721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.885244 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.885721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.885274 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.885721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.885321 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.885721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.885364 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.891673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.891642 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.893008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.892982 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.893401 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.893383 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.894986 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895060 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949r8\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-kube-api-access-949r8\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895117 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895194 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895454 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895540 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29a321b-415d-413f-9d6e-d38150ef8e67-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895599 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895653 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895686 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29swq\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-kube-api-access-29swq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895781 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.895979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.895809 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.897457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.897119 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.897457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.897167 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29a321b-415d-413f-9d6e-d38150ef8e67-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.904753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.903791 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.905766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.904938 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz"] Nov 03 19:03:49.906616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.906588 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.994582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.994496 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-949r8\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-kube-api-access-949r8\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:49.998944 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.998907 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999093 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999134 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999196 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999248 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29swq\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-kube-api-access-29swq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999277 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999441 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999516 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:49.999605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:49.999583 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.000770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.000687 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.001007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.000982 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.001808 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.001759 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.002601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.002578 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.002854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.002839 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.004989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.004964 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.005553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.005529 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:50.006029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.006000 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.032754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.032508 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:50.040298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.040268 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:50.040404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.040315 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:50.078337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.077882 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29swq\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-kube-api-access-29swq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.078337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.078292 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.114004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.113900 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:50.597493 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.597432 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt"] Nov 03 19:03:50.618831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.604336 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:50.618831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.604429 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:50.618831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.604484 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:50.696690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.696653 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz"] Nov 03 19:03:50.921946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.840210 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:50.921946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.912604 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:50.921946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.912966 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:03:50.935756 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:03:50.933959 4948 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29a321b_415d_413f_9d6e_d38150ef8e67.slice/crio-0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:03:50.945473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.945421 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" event={"ID":"cd15b0fc-c46e-438c-a583-0e69d0dacb5d","Type":"ContainerStarted","Data":"32d7f1d4407272545a82871d442e64e1037e5e393cf17d805bd5b726c8eacc34"} Nov 03 19:03:50.947310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:50.947270 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" event={"ID":"f29a321b-415d-413f-9d6e-d38150ef8e67","Type":"ContainerStarted","Data":"a10b78f4077a908b239cd08ab745aad784351434dbf97aca5d63a31ad4f85c98"} Nov 03 19:03:51.041556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:51.040920 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:51.041556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:51.040969 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:52.003724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:51.993923 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" event={"ID":"cd15b0fc-c46e-438c-a583-0e69d0dacb5d","Type":"ContainerStarted","Data":"a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda"} Nov 03 19:03:52.007722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.004636 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" event={"ID":"f29a321b-415d-413f-9d6e-d38150ef8e67","Type":"ContainerStarted","Data":"0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d"} Nov 03 19:03:52.033829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.033790 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:03:52.034816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.034786 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:52.034915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.034843 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:52.040523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.040486 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:52.040693 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.040674 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:52.134722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.134343 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:03:52.134722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.134537 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:52.134722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.134585 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:52.137723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.136228 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podStartSLOduration=3.136210968 podStartE2EDuration="3.136210968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:52.117664131 +0000 UTC m=+1561.600660046" watchObservedRunningTime="2026-11-03 19:03:52.136210968 +0000 UTC m=+1561.619206883" Nov 03 19:03:52.245719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:52.244671 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podStartSLOduration=3.244653836 podStartE2EDuration="3.244653836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:52.242991953 +0000 UTC m=+1561.725987870" watchObservedRunningTime="2026-11-03 19:03:52.244653836 +0000 UTC m=+1561.727649751" Nov 03 19:03:53.049971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:53.048151 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:53.049971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:53.048212 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:53.049971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:53.049029 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:53.049971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:53.049067 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:53.129687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:53.129650 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:53.129990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:53.129956 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:54.034883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.034842 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:54.072828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.072778 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:54.074261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.040165 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:54.074491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.074462 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:54.141517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.141479 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:54.142256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.142223 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:54.630205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.630162 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8"] Nov 03 19:03:54.638170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.638143 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.643871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.643835 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8"] Nov 03 19:03:54.709163 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.709112 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:03:54.710537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.710513 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:03:54.723736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.721462 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:03:54.723736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.721524 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:03:54.837996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.837952 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.838188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.838011 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdt6t\" (UniqueName: \"kubernetes.io/projected/0926fd17-9755-47fa-9784-c1189d6922a2-kube-api-access-fdt6t\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.838188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.838169 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.940729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.939974 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.940729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.940015 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdt6t\" (UniqueName: \"kubernetes.io/projected/0926fd17-9755-47fa-9784-c1189d6922a2-kube-api-access-fdt6t\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.940729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.940184 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.941845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.941821 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:54.943405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:54.943382 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:55.030632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.030588 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdt6t\" (UniqueName: \"kubernetes.io/projected/0926fd17-9755-47fa-9784-c1189d6922a2-kube-api-access-fdt6t\") pod \"maas-api-revoke-keys-e2e-ait-90774e61-5qmb8\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:55.033316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.033287 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:55.033388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.033347 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:55.042711 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.042612 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:55.042711 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.042667 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:55.115613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.115341 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:55.115613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.115403 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:55.281478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:55.281392 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:56.033238 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.033203 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:56.033601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.033552 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:56.040987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.040964 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 19:03:56.041116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.041031 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 19:03:56.120754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.115188 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:56.120754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.115264 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:56.125389 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:03:56.124955 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0926fd17_9755_47fa_9784_c1189d6922a2.slice/crio-ad137338332dbbe1e0c60b7b38a62c63152efa9271b8a52ac319218a0e126e6a WatchSource:0}: Error finding container ad137338332dbbe1e0c60b7b38a62c63152efa9271b8a52ac319218a0e126e6a: Status 404 returned error can't find the container with id ad137338332dbbe1e0c60b7b38a62c63152efa9271b8a52ac319218a0e126e6a Nov 03 19:03:56.129359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.129326 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8"] Nov 03 19:03:56.221002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.220956 4948 scope.go:122] "RemoveContainer" containerID="6e64c05e3cbc2be8b480ca6ef888e0b71d75a94666b0f8962af14fc93410e62e" Nov 03 19:03:56.246898 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.246872 4948 scope.go:122] "RemoveContainer" containerID="f6f06a7b2569da48544ec59d1c4249da5c1a7cc6e6656b0436b5d838c96198a9" Nov 03 19:03:56.351740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.344547 4948 scope.go:122] "RemoveContainer" containerID="cec95e403f993f200356c0eb53de9b3faed438835642bc2f2f13fee76dc51926" Nov 03 19:03:56.473688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.473593 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" event={"ID":"0926fd17-9755-47fa-9784-c1189d6922a2","Type":"ContainerStarted","Data":"ad137338332dbbe1e0c60b7b38a62c63152efa9271b8a52ac319218a0e126e6a"} Nov 03 19:03:56.534943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.533899 4948 scope.go:122] "RemoveContainer" containerID="b1aa4321ae5130682a024410ce73ed0aa8db211a9e9fd2118a0c74d95aae7f48" Nov 03 19:03:56.702794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.687944 4948 scope.go:122] "RemoveContainer" containerID="261aa9ab6ed72ece0cd33eb2507b19688bcc5fd67d5b93eaac06d9eb661eaf74" Nov 03 19:03:56.810979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.810406 4948 scope.go:122] "RemoveContainer" containerID="a478a99c0c3faf7871520a7712c764023551699d98572d9cdd60a2ef86113183" Nov 03 19:03:56.897623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:56.897232 4948 scope.go:122] "RemoveContainer" containerID="f4f16f6179c4eb8f31eca8d251831c86187f9f6bd5d879031483e5e49d5af4fa" Nov 03 19:03:57.039153 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:57.039062 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:57.039153 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:57.039114 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:57.150729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:57.147832 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:57.150729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:57.147906 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:58.048724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.043822 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:58.048724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.043880 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:58.097968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.090825 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:03:58.097968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.090902 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:58.115739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.109183 4948 generic.go:356] "Generic (PLEG): container finished" podID="0926fd17-9755-47fa-9784-c1189d6922a2" containerID="8f1e732747027d6091028d2aad6433c599c09dbd882bfdd674e32bf3ef2738fd" exitCode=0 Nov 03 19:03:58.115739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.109303 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" event={"ID":"0926fd17-9755-47fa-9784-c1189d6922a2","Type":"ContainerDied","Data":"8f1e732747027d6091028d2aad6433c599c09dbd882bfdd674e32bf3ef2738fd"} Nov 03 19:03:58.121742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.117278 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:58.121742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:58.117338 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:59.013198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.013161 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:59.014224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.014195 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:59.015392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.015369 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:03:59.033365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.033331 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:03:59.033600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.033567 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:03:59.114378 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.114339 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 19:03:59.114593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.114405 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 19:03:59.367739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.364128 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:03:59.453859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.453820 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-xks-ca\") pod \"0926fd17-9755-47fa-9784-c1189d6922a2\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " Nov 03 19:03:59.454544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.454514 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-openshift-ca\") pod \"0926fd17-9755-47fa-9784-c1189d6922a2\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " Nov 03 19:03:59.454983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.454961 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0926fd17-9755-47fa-9784-c1189d6922a2-kube-api-access-fdt6t\" (UniqueName: \"kubernetes.io/projected/0926fd17-9755-47fa-9784-c1189d6922a2-kube-api-access-fdt6t\") pod \"0926fd17-9755-47fa-9784-c1189d6922a2\" (UID: \"0926fd17-9755-47fa-9784-c1189d6922a2\") " Nov 03 19:03:59.455208 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.455181 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-openshift-ca" pod "0926fd17-9755-47fa-9784-c1189d6922a2" (UID: "0926fd17-9755-47fa-9784-c1189d6922a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:03:59.455981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.455955 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:03:59.456428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.456401 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-xks-ca" pod "0926fd17-9755-47fa-9784-c1189d6922a2" (UID: "0926fd17-9755-47fa-9784-c1189d6922a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:03:59.482771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.479714 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0926fd17-9755-47fa-9784-c1189d6922a2-kube-api-access-fdt6t" pod "0926fd17-9755-47fa-9784-c1189d6922a2" (UID: "0926fd17-9755-47fa-9784-c1189d6922a2"). InnerVolumeSpecName "kube-api-access-fdt6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:03:59.558527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.558471 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0926fd17-9755-47fa-9784-c1189d6922a2-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:03:59.558527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:03:59.558533 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdt6t\" (UniqueName: \"kubernetes.io/projected/0926fd17-9755-47fa-9784-c1189d6922a2-kube-api-access-fdt6t\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:00.145715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.145398 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" Nov 03 19:04:00.145715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.145442 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8" event={"ID":"0926fd17-9755-47fa-9784-c1189d6922a2","Type":"ContainerDied","Data":"ad137338332dbbe1e0c60b7b38a62c63152efa9271b8a52ac319218a0e126e6a"} Nov 03 19:04:00.145715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.145509 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad137338332dbbe1e0c60b7b38a62c63152efa9271b8a52ac319218a0e126e6a" Nov 03 19:04:00.200761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.200475 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:04:00.201455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.201424 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:04:00.202517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.202492 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:04:00.383749 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.375990 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:04:00.383749 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.377693 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:04:00.395085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:00.395055 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:04:02.806369 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:04:02.806274 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:04:06.143421 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:06.143385 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt"] Nov 03 19:04:06.277801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:06.277757 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g"] Nov 03 19:04:10.255332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.255291 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6"] Nov 03 19:04:10.256240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.256219 4948 cpu_manager.go:414] "containerMap: removing container" podUID="0926fd17-9755-47fa-9784-c1189d6922a2" containerName="revoke-keys" Nov 03 19:04:10.256240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.256240 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0926fd17-9755-47fa-9784-c1189d6922a2" containerName="revoke-keys" Nov 03 19:04:10.256398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.256365 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="0926fd17-9755-47fa-9784-c1189d6922a2" containerName="revoke-keys" Nov 03 19:04:10.260509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.260485 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.290072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.290039 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6th7q\" (UniqueName: \"kubernetes.io/projected/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-kube-api-access-6th7q\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.290199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.290153 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.290574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.290550 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.313057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.313027 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6"] Nov 03 19:04:10.357587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.357557 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:04:10.357834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.357604 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:04:10.358274 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.358255 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:04:10.358404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.358362 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:04:10.392993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.392956 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.393168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.393112 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6th7q\" (UniqueName: \"kubernetes.io/projected/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-kube-api-access-6th7q\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.393239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.393172 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.393806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.393783 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.395627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.395603 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.447322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.447275 4948 generic.go:356] "Generic (PLEG): container finished" podID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerID="dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350" exitCode=0 Nov 03 19:04:10.447759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.447730 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" event={"ID":"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a","Type":"ContainerDied","Data":"dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350"} Nov 03 19:04:10.448400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.448376 4948 scope.go:122] "RemoveContainer" containerID="dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350" Nov 03 19:04:10.450590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.450559 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:10.450688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.450636 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:10.450688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.450668 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:10.455794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.455769 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6th7q\" (UniqueName: \"kubernetes.io/projected/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-kube-api-access-6th7q\") pod \"maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:10.575582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:10.575550 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:11.003306 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:04:11.003269 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfe0a7d_90c1_4cf2_ae81_29ca81f0bd6c.slice/crio-4adbe9c0054b3fdc60e5e82e5dd9e8019f97d1db1b26352579a4c8dd02f09dae WatchSource:0}: Error finding container 4adbe9c0054b3fdc60e5e82e5dd9e8019f97d1db1b26352579a4c8dd02f09dae: Status 404 returned error can't find the container with id 4adbe9c0054b3fdc60e5e82e5dd9e8019f97d1db1b26352579a4c8dd02f09dae Nov 03 19:04:11.006402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.006377 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6"] Nov 03 19:04:11.144533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.144490 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" podUID="2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" containerName="payload-processing" containerID="cri-o://4bd5a8bc6f2294a43ad7fc3772caff4c82a32fe0c658beb8e1b5419611098988" gracePeriod=25 Nov 03 19:04:11.282281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.282189 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" podUID="4b51d6f9-dbec-4fa0-a0b3-072746a6485d" containerName="payload-pre-processing" containerID="cri-o://76457c297db8c85ddaa9cd9f762b317678719d5a89fb2b1faff101472511e29f" gracePeriod=25 Nov 03 19:04:11.519189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.519148 4948 generic.go:356] "Generic (PLEG): container finished" podID="2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" containerID="4bd5a8bc6f2294a43ad7fc3772caff4c82a32fe0c658beb8e1b5419611098988" exitCode=0 Nov 03 19:04:11.519360 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.519253 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" event={"ID":"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c","Type":"ContainerDied","Data":"4bd5a8bc6f2294a43ad7fc3772caff4c82a32fe0c658beb8e1b5419611098988"} Nov 03 19:04:11.521950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.521875 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" event={"ID":"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c","Type":"ContainerStarted","Data":"4adbe9c0054b3fdc60e5e82e5dd9e8019f97d1db1b26352579a4c8dd02f09dae"} Nov 03 19:04:11.525972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.525944 4948 generic.go:356] "Generic (PLEG): container finished" podID="4b51d6f9-dbec-4fa0-a0b3-072746a6485d" containerID="76457c297db8c85ddaa9cd9f762b317678719d5a89fb2b1faff101472511e29f" exitCode=0 Nov 03 19:04:11.546101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.526231 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" event={"ID":"4b51d6f9-dbec-4fa0-a0b3-072746a6485d","Type":"ContainerDied","Data":"76457c297db8c85ddaa9cd9f762b317678719d5a89fb2b1faff101472511e29f"} Nov 03 19:04:11.548501 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.548478 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:04:11.606665 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.606627 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" event={"ID":"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a","Type":"ContainerStarted","Data":"a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d"} Nov 03 19:04:11.625867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.625831 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-plugins-config-volume\") pod \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " Nov 03 19:04:11.626050 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.625903 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-kube-api-access-x8f77\" (UniqueName: \"kubernetes.io/projected/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-kube-api-access-x8f77\") pod \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\" (UID: \"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c\") " Nov 03 19:04:11.626592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.626564 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-plugins-config-volume" pod "2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" (UID: "2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:11.629599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.629567 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-kube-api-access-x8f77" pod "2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" (UID: "2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c"). InnerVolumeSpecName "kube-api-access-x8f77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:11.731143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.731091 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:11.731143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.731127 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8f77\" (UniqueName: \"kubernetes.io/projected/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c-kube-api-access-x8f77\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:11.790831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.790777 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:04:11.799811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.791117 4948 patch_prober.go:29] interesting pod/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:04:11.799811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.791174 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:04:11.850197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.850172 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:04:11.933017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.932876 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-plugins-config-volume\") pod \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " Nov 03 19:04:11.933230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.933142 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-kube-api-access-wjg7w\" (UniqueName: \"kubernetes.io/projected/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-kube-api-access-wjg7w\") pod \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\" (UID: \"4b51d6f9-dbec-4fa0-a0b3-072746a6485d\") " Nov 03 19:04:11.933321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.933284 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-plugins-config-volume" pod "4b51d6f9-dbec-4fa0-a0b3-072746a6485d" (UID: "4b51d6f9-dbec-4fa0-a0b3-072746a6485d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:11.933853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.933818 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:11.935397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:11.935367 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-kube-api-access-wjg7w" pod "4b51d6f9-dbec-4fa0-a0b3-072746a6485d" (UID: "4b51d6f9-dbec-4fa0-a0b3-072746a6485d"). InnerVolumeSpecName "kube-api-access-wjg7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:12.035341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.035306 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjg7w\" (UniqueName: \"kubernetes.io/projected/4b51d6f9-dbec-4fa0-a0b3-072746a6485d-kube-api-access-wjg7w\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:12.788721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.779576 4948 patch_prober.go:29] interesting pod/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:04:12.789327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.789288 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:04:12.938607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.938573 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" Nov 03 19:04:12.939444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.939414 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g" event={"ID":"4b51d6f9-dbec-4fa0-a0b3-072746a6485d","Type":"ContainerDied","Data":"a66d3010fbf223016adf981cbe3336be86803ab69a276488ed932691ca822828"} Nov 03 19:04:12.939606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.939471 4948 scope.go:122] "RemoveContainer" containerID="76457c297db8c85ddaa9cd9f762b317678719d5a89fb2b1faff101472511e29f" Nov 03 19:04:12.940657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.940630 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" event={"ID":"2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c","Type":"ContainerDied","Data":"009f0ce9b7d490eafd0b55aee5ccfa657981e7598d87d075d529cd4241226afa"} Nov 03 19:04:12.940928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.940894 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt" Nov 03 19:04:12.944086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.944064 4948 generic.go:356] "Generic (PLEG): container finished" podID="6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" containerID="3fac942a7054f8849937ee718331fcad60a0306585ad7dfefaa1611772db7b83" exitCode=0 Nov 03 19:04:12.944835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.944812 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" event={"ID":"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c","Type":"ContainerDied","Data":"3fac942a7054f8849937ee718331fcad60a0306585ad7dfefaa1611772db7b83"} Nov 03 19:04:12.992975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:12.988509 4948 scope.go:122] "RemoveContainer" containerID="4bd5a8bc6f2294a43ad7fc3772caff4c82a32fe0c658beb8e1b5419611098988" Nov 03 19:04:13.111471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:13.111387 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g"] Nov 03 19:04:13.188745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:13.188690 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90774e61-5d6679746c-zph6g"] Nov 03 19:04:13.298782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:13.298750 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt"] Nov 03 19:04:13.305150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:13.305116 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90774e61-5758cd8cfc-mstxt"] Nov 03 19:04:13.767898 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:13.767867 4948 patch_prober.go:29] interesting pod/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:04:13.768176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:13.768149 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:04:14.239231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.239203 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:14.309126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.309033 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-xks-ca\") pod \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " Nov 03 19:04:14.309126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.309086 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-openshift-ca\") pod \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " Nov 03 19:04:14.309408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.309387 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-kube-api-access-6th7q\" (UniqueName: \"kubernetes.io/projected/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-kube-api-access-6th7q\") pod \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\" (UID: \"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c\") " Nov 03 19:04:14.309489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.309466 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-openshift-ca" pod "6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" (UID: "6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:14.310050 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.310027 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:14.311301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.311276 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-xks-ca" pod "6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" (UID: "6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:14.311450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.311426 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-kube-api-access-6th7q" pod "6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" (UID: "6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c"). InnerVolumeSpecName "kube-api-access-6th7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:14.411439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.411411 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6th7q\" (UniqueName: \"kubernetes.io/projected/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-kube-api-access-6th7q\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:14.411439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.411438 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:14.763723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.763430 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" path="/var/lib/kubelet/pods/2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c/volumes" Nov 03 19:04:14.780723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.764194 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b51d6f9-dbec-4fa0-a0b3-072746a6485d" path="/var/lib/kubelet/pods/4b51d6f9-dbec-4fa0-a0b3-072746a6485d/volumes" Nov 03 19:04:14.780723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.768807 4948 patch_prober.go:29] interesting pod/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:04:14.780723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.769144 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:04:14.999744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.999610 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" Nov 03 19:04:14.999744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.999632 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6" event={"ID":"6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c","Type":"ContainerDied","Data":"4adbe9c0054b3fdc60e5e82e5dd9e8019f97d1db1b26352579a4c8dd02f09dae"} Nov 03 19:04:14.999744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:14.999685 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4adbe9c0054b3fdc60e5e82e5dd9e8019f97d1db1b26352579a4c8dd02f09dae" Nov 03 19:04:15.385841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:15.385805 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:04:15.772072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:15.772037 4948 patch_prober.go:29] interesting pod/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:04:15.772374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:15.772339 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:04:15.977693 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:15.977625 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27"] Nov 03 19:04:16.013289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.013251 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" containerName="payload-processing" Nov 03 19:04:16.013547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.013527 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" containerName="payload-processing" Nov 03 19:04:16.013816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.013797 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4b51d6f9-dbec-4fa0-a0b3-072746a6485d" containerName="payload-pre-processing" Nov 03 19:04:16.014052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.014035 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b51d6f9-dbec-4fa0-a0b3-072746a6485d" containerName="payload-pre-processing" Nov 03 19:04:16.014190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.014178 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" containerName="revoke-keys" Nov 03 19:04:16.014277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.014266 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" containerName="revoke-keys" Nov 03 19:04:16.014799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.014779 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" containerName="revoke-keys" Nov 03 19:04:16.014930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.014917 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2d1c8b-8fe9-4e6e-80c6-926d59697f0c" containerName="payload-processing" Nov 03 19:04:16.015056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.015042 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b51d6f9-dbec-4fa0-a0b3-072746a6485d" containerName="payload-pre-processing" Nov 03 19:04:16.022471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.022402 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27"] Nov 03 19:04:16.022579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.022525 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.045184 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.045162 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjt8z\" (UniqueName: \"kubernetes.io/projected/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-kube-api-access-sjt8z\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.045320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.045307 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.045459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.045447 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.086292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.086244 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:04:16.137030 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.137000 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:04:16.137167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.137067 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:04:16.137167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.137095 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:04:16.147043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.147018 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjt8z\" (UniqueName: \"kubernetes.io/projected/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-kube-api-access-sjt8z\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.147168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.147051 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.147168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.147114 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.147874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.147852 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.149525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.149500 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.210272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.210233 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjt8z\" (UniqueName: \"kubernetes.io/projected/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-kube-api-access-sjt8z\") pod \"maas-api-revoke-keys-e2e-derive-30b92815-l6k27\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.345113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.345029 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:16.771167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.771130 4948 patch_prober.go:29] interesting pod/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:04:16.771809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.771767 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:04:16.793802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:16.792895 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27"] Nov 03 19:04:16.793802 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:04:16.793384 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a09d5f_8abd_4d07_a780_a01e7bc17c39.slice/crio-1d14a41122a8097eeaa05cbcbc66f96be30bab9833ee15ddd9002e44c9a8c78d WatchSource:0}: Error finding container 1d14a41122a8097eeaa05cbcbc66f96be30bab9833ee15ddd9002e44c9a8c78d: Status 404 returned error can't find the container with id 1d14a41122a8097eeaa05cbcbc66f96be30bab9833ee15ddd9002e44c9a8c78d Nov 03 19:04:17.037719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:17.037602 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" event={"ID":"c4a09d5f-8abd-4d07-a780-a01e7bc17c39","Type":"ContainerStarted","Data":"1d14a41122a8097eeaa05cbcbc66f96be30bab9833ee15ddd9002e44c9a8c78d"} Nov 03 19:04:17.770686 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:17.770660 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:04:17.771671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:17.771649 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:04:18.069406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:18.069317 4948 generic.go:356] "Generic (PLEG): container finished" podID="c4a09d5f-8abd-4d07-a780-a01e7bc17c39" containerID="aa4f44ad9f91d34c8200fc8117d93c4f79d1ee0373914baf8517f2b8da4de5cf" exitCode=0 Nov 03 19:04:18.070033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:18.070001 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" event={"ID":"c4a09d5f-8abd-4d07-a780-a01e7bc17c39","Type":"ContainerDied","Data":"aa4f44ad9f91d34c8200fc8117d93c4f79d1ee0373914baf8517f2b8da4de5cf"} Nov 03 19:04:19.253826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.253801 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:19.333883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.333803 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-openshift-ca\") pod \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " Nov 03 19:04:19.334038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.333921 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-xks-ca\") pod \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " Nov 03 19:04:19.334079 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.334049 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-kube-api-access-sjt8z\" (UniqueName: \"kubernetes.io/projected/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-kube-api-access-sjt8z\") pod \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\" (UID: \"c4a09d5f-8abd-4d07-a780-a01e7bc17c39\") " Nov 03 19:04:19.334226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.334205 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-openshift-ca" pod "c4a09d5f-8abd-4d07-a780-a01e7bc17c39" (UID: "c4a09d5f-8abd-4d07-a780-a01e7bc17c39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:19.334796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.334773 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:19.336099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.336080 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-xks-ca" pod "c4a09d5f-8abd-4d07-a780-a01e7bc17c39" (UID: "c4a09d5f-8abd-4d07-a780-a01e7bc17c39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:19.336196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.336174 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-kube-api-access-sjt8z" pod "c4a09d5f-8abd-4d07-a780-a01e7bc17c39" (UID: "c4a09d5f-8abd-4d07-a780-a01e7bc17c39"). InnerVolumeSpecName "kube-api-access-sjt8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:19.436653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.436617 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:19.436653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:19.436648 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjt8z\" (UniqueName: \"kubernetes.io/projected/c4a09d5f-8abd-4d07-a780-a01e7bc17c39-kube-api-access-sjt8z\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:20.090417 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:20.090387 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" Nov 03 19:04:20.090614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:20.090424 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27" event={"ID":"c4a09d5f-8abd-4d07-a780-a01e7bc17c39","Type":"ContainerDied","Data":"1d14a41122a8097eeaa05cbcbc66f96be30bab9833ee15ddd9002e44c9a8c78d"} Nov 03 19:04:20.090614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:20.090489 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d14a41122a8097eeaa05cbcbc66f96be30bab9833ee15ddd9002e44c9a8c78d" Nov 03 19:04:21.906248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:21.906213 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x"] Nov 03 19:04:25.055166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:25.055130 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg"] Nov 03 19:04:25.105995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:25.105957 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f011ae-zcwtg"] Nov 03 19:04:26.755833 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:26.755798 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a9dc4c-5b14-4d02-8605-3d7501316903" path="/var/lib/kubelet/pods/44a9dc4c-5b14-4d02-8605-3d7501316903/volumes" Nov 03 19:04:26.906600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:26.906561 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" podUID="4eec5248-4944-46a7-bff9-5da030569f0f" containerName="payload-pre-processing" containerID="cri-o://10dee66c90bcd35242de21739908f926ce49594d392bfd6dbd4542625b9efa3c" gracePeriod=25 Nov 03 19:04:27.171682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.171645 4948 generic.go:356] "Generic (PLEG): container finished" podID="4eec5248-4944-46a7-bff9-5da030569f0f" containerID="10dee66c90bcd35242de21739908f926ce49594d392bfd6dbd4542625b9efa3c" exitCode=0 Nov 03 19:04:27.171836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.171782 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" event={"ID":"4eec5248-4944-46a7-bff9-5da030569f0f","Type":"ContainerDied","Data":"10dee66c90bcd35242de21739908f926ce49594d392bfd6dbd4542625b9efa3c"} Nov 03 19:04:27.207527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.207495 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:04:27.331847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.331770 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eec5248-4944-46a7-bff9-5da030569f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eec5248-4944-46a7-bff9-5da030569f0f-plugins-config-volume\") pod \"4eec5248-4944-46a7-bff9-5da030569f0f\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " Nov 03 19:04:27.332008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.331990 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eec5248-4944-46a7-bff9-5da030569f0f-kube-api-access-fljds\" (UniqueName: \"kubernetes.io/projected/4eec5248-4944-46a7-bff9-5da030569f0f-kube-api-access-fljds\") pod \"4eec5248-4944-46a7-bff9-5da030569f0f\" (UID: \"4eec5248-4944-46a7-bff9-5da030569f0f\") " Nov 03 19:04:27.332240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.332213 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eec5248-4944-46a7-bff9-5da030569f0f-plugins-config-volume" pod "4eec5248-4944-46a7-bff9-5da030569f0f" (UID: "4eec5248-4944-46a7-bff9-5da030569f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:27.332717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.332674 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eec5248-4944-46a7-bff9-5da030569f0f-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:27.334224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.334197 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eec5248-4944-46a7-bff9-5da030569f0f-kube-api-access-fljds" pod "4eec5248-4944-46a7-bff9-5da030569f0f" (UID: "4eec5248-4944-46a7-bff9-5da030569f0f"). InnerVolumeSpecName "kube-api-access-fljds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:27.433922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.433889 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fljds\" (UniqueName: \"kubernetes.io/projected/4eec5248-4944-46a7-bff9-5da030569f0f-kube-api-access-fljds\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:27.463607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.463575 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh"] Nov 03 19:04:27.585680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:27.585578 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c"] Nov 03 19:04:28.183871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:28.183833 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" event={"ID":"4eec5248-4944-46a7-bff9-5da030569f0f","Type":"ContainerDied","Data":"76ed12db59eb77db990fc2d7e4311ff3bf2c3122512cfdf1f15897d821f59ebe"} Nov 03 19:04:28.184337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:28.183911 4948 scope.go:122] "RemoveContainer" containerID="10dee66c90bcd35242de21739908f926ce49594d392bfd6dbd4542625b9efa3c" Nov 03 19:04:28.184337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:28.183843 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x" Nov 03 19:04:28.403466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:28.403424 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x"] Nov 03 19:04:28.408788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:28.408762 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f198bfb-587965948c-z9n4x"] Nov 03 19:04:28.756334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:28.756302 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eec5248-4944-46a7-bff9-5da030569f0f" path="/var/lib/kubelet/pods/4eec5248-4944-46a7-bff9-5da030569f0f/volumes" Nov 03 19:04:29.114920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.114838 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4"] Nov 03 19:04:29.116193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.116163 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c4a09d5f-8abd-4d07-a780-a01e7bc17c39" containerName="revoke-keys" Nov 03 19:04:29.116334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.116322 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a09d5f-8abd-4d07-a780-a01e7bc17c39" containerName="revoke-keys" Nov 03 19:04:29.116453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.116441 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4eec5248-4944-46a7-bff9-5da030569f0f" containerName="payload-pre-processing" Nov 03 19:04:29.116536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.116525 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eec5248-4944-46a7-bff9-5da030569f0f" containerName="payload-pre-processing" Nov 03 19:04:29.116915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.116895 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a09d5f-8abd-4d07-a780-a01e7bc17c39" containerName="revoke-keys" Nov 03 19:04:29.117047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.117035 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eec5248-4944-46a7-bff9-5da030569f0f" containerName="payload-pre-processing" Nov 03 19:04:29.128380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.128352 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.192730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.190860 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4"] Nov 03 19:04:29.192730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.191147 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:04:29.192730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.191261 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:04:29.192730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.191803 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:04:29.192730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.192396 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:04:29.608720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.590746 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzbq\" (UniqueName: \"kubernetes.io/projected/049a88a5-7b87-4324-9b8c-ea46d263756f-kube-api-access-brzbq\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.663397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.662885 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.663397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.663011 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.765390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.765349 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.767849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.767819 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.769272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.769250 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzbq\" (UniqueName: \"kubernetes.io/projected/049a88a5-7b87-4324-9b8c-ea46d263756f-kube-api-access-brzbq\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.769883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.768958 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.811751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.809963 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.846754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.846681 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzbq\" (UniqueName: \"kubernetes.io/projected/049a88a5-7b87-4324-9b8c-ea46d263756f-kube-api-access-brzbq\") pod \"maas-api-revoke-sub-d1ac4a0a-662n4\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:29.965067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:29.965034 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:30.361685 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:04:30.361651 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049a88a5_7b87_4324_9b8c_ea46d263756f.slice/crio-a62b494cbc8f0cbe000ace008fd7f16c822b5c6ff1c59ad91e2a0b063b925b63 WatchSource:0}: Error finding container a62b494cbc8f0cbe000ace008fd7f16c822b5c6ff1c59ad91e2a0b063b925b63: Status 404 returned error can't find the container with id a62b494cbc8f0cbe000ace008fd7f16c822b5c6ff1c59ad91e2a0b063b925b63 Nov 03 19:04:30.363714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:30.363677 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4"] Nov 03 19:04:30.689775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:30.689735 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" event={"ID":"049a88a5-7b87-4324-9b8c-ea46d263756f","Type":"ContainerStarted","Data":"a62b494cbc8f0cbe000ace008fd7f16c822b5c6ff1c59ad91e2a0b063b925b63"} Nov 03 19:04:31.701785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:31.701750 4948 generic.go:356] "Generic (PLEG): container finished" podID="049a88a5-7b87-4324-9b8c-ea46d263756f" containerID="96423b30a93111d9cacf217466755bf20accdcb126640457d8dcdd041fbc7761" exitCode=0 Nov 03 19:04:31.702190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:31.701817 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" event={"ID":"049a88a5-7b87-4324-9b8c-ea46d263756f","Type":"ContainerDied","Data":"96423b30a93111d9cacf217466755bf20accdcb126640457d8dcdd041fbc7761"} Nov 03 19:04:32.464923 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.464876 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" podUID="b2de5c44-10cd-46c3-8386-25ad4b135c5d" containerName="payload-processing" containerID="cri-o://ac68b77d12a7c8f8c05ee7bdf48309437c2ce9409029943ceb14c1a96e116c85" gracePeriod=25 Nov 03 19:04:32.586514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.586439 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" podUID="2a226ada-8eab-4c72-b8f4-26e476e9dece" containerName="payload-pre-processing" containerID="cri-o://3f7f5995a87da8fd231741f1927a9cf884673cae09a00bda707c92e1af792145" gracePeriod=25 Nov 03 19:04:32.721292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.720935 4948 generic.go:356] "Generic (PLEG): container finished" podID="b2de5c44-10cd-46c3-8386-25ad4b135c5d" containerID="ac68b77d12a7c8f8c05ee7bdf48309437c2ce9409029943ceb14c1a96e116c85" exitCode=0 Nov 03 19:04:32.721292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.721189 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" event={"ID":"b2de5c44-10cd-46c3-8386-25ad4b135c5d","Type":"ContainerDied","Data":"ac68b77d12a7c8f8c05ee7bdf48309437c2ce9409029943ceb14c1a96e116c85"} Nov 03 19:04:32.723286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.723261 4948 generic.go:356] "Generic (PLEG): container finished" podID="2a226ada-8eab-4c72-b8f4-26e476e9dece" containerID="3f7f5995a87da8fd231741f1927a9cf884673cae09a00bda707c92e1af792145" exitCode=0 Nov 03 19:04:32.723657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.723635 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" event={"ID":"2a226ada-8eab-4c72-b8f4-26e476e9dece","Type":"ContainerDied","Data":"3f7f5995a87da8fd231741f1927a9cf884673cae09a00bda707c92e1af792145"} Nov 03 19:04:32.783900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.783863 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:04:32.820887 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:04:32.820847 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:04:32.916718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.915877 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2de5c44-10cd-46c3-8386-25ad4b135c5d-kube-api-access-7qd98\" (UniqueName: \"kubernetes.io/projected/b2de5c44-10cd-46c3-8386-25ad4b135c5d-kube-api-access-7qd98\") pod \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " Nov 03 19:04:32.916718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.916196 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2de5c44-10cd-46c3-8386-25ad4b135c5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de5c44-10cd-46c3-8386-25ad4b135c5d-plugins-config-volume\") pod \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\" (UID: \"b2de5c44-10cd-46c3-8386-25ad4b135c5d\") " Nov 03 19:04:32.929726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.917208 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2de5c44-10cd-46c3-8386-25ad4b135c5d-plugins-config-volume" pod "b2de5c44-10cd-46c3-8386-25ad4b135c5d" (UID: "b2de5c44-10cd-46c3-8386-25ad4b135c5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:32.978773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:32.968199 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2de5c44-10cd-46c3-8386-25ad4b135c5d-kube-api-access-7qd98" pod "b2de5c44-10cd-46c3-8386-25ad4b135c5d" (UID: "b2de5c44-10cd-46c3-8386-25ad4b135c5d"). InnerVolumeSpecName "kube-api-access-7qd98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:33.025725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.023087 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de5c44-10cd-46c3-8386-25ad4b135c5d-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:33.025725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.023127 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qd98\" (UniqueName: \"kubernetes.io/projected/b2de5c44-10cd-46c3-8386-25ad4b135c5d-kube-api-access-7qd98\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:33.086406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.086375 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:04:33.103459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.103370 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:33.225203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.225168 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-xks-ca\") pod \"049a88a5-7b87-4324-9b8c-ea46d263756f\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " Nov 03 19:04:33.225203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.225207 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049a88a5-7b87-4324-9b8c-ea46d263756f-kube-api-access-brzbq\" (UniqueName: \"kubernetes.io/projected/049a88a5-7b87-4324-9b8c-ea46d263756f-kube-api-access-brzbq\") pod \"049a88a5-7b87-4324-9b8c-ea46d263756f\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " Nov 03 19:04:33.225462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.225230 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-openshift-ca\") pod \"049a88a5-7b87-4324-9b8c-ea46d263756f\" (UID: \"049a88a5-7b87-4324-9b8c-ea46d263756f\") " Nov 03 19:04:33.225462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.225418 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a226ada-8eab-4c72-b8f4-26e476e9dece-kube-api-access-9qhd2\" (UniqueName: \"kubernetes.io/projected/2a226ada-8eab-4c72-b8f4-26e476e9dece-kube-api-access-9qhd2\") pod \"2a226ada-8eab-4c72-b8f4-26e476e9dece\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " Nov 03 19:04:33.225582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.225487 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a226ada-8eab-4c72-b8f4-26e476e9dece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a226ada-8eab-4c72-b8f4-26e476e9dece-plugins-config-volume\") pod \"2a226ada-8eab-4c72-b8f4-26e476e9dece\" (UID: \"2a226ada-8eab-4c72-b8f4-26e476e9dece\") " Nov 03 19:04:33.225738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.225682 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-openshift-ca" pod "049a88a5-7b87-4324-9b8c-ea46d263756f" (UID: "049a88a5-7b87-4324-9b8c-ea46d263756f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:33.225917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.225889 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a226ada-8eab-4c72-b8f4-26e476e9dece-plugins-config-volume" pod "2a226ada-8eab-4c72-b8f4-26e476e9dece" (UID: "2a226ada-8eab-4c72-b8f4-26e476e9dece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:33.226347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.226319 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a226ada-8eab-4c72-b8f4-26e476e9dece-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:33.226450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.226353 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:33.227523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.227495 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-xks-ca" pod "049a88a5-7b87-4324-9b8c-ea46d263756f" (UID: "049a88a5-7b87-4324-9b8c-ea46d263756f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:33.227669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.227651 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a226ada-8eab-4c72-b8f4-26e476e9dece-kube-api-access-9qhd2" pod "2a226ada-8eab-4c72-b8f4-26e476e9dece" (UID: "2a226ada-8eab-4c72-b8f4-26e476e9dece"). InnerVolumeSpecName "kube-api-access-9qhd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:33.227836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.227818 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049a88a5-7b87-4324-9b8c-ea46d263756f-kube-api-access-brzbq" pod "049a88a5-7b87-4324-9b8c-ea46d263756f" (UID: "049a88a5-7b87-4324-9b8c-ea46d263756f"). InnerVolumeSpecName "kube-api-access-brzbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:33.328358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.328261 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a88a5-7b87-4324-9b8c-ea46d263756f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:33.328358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.328299 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brzbq\" (UniqueName: \"kubernetes.io/projected/049a88a5-7b87-4324-9b8c-ea46d263756f-kube-api-access-brzbq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:33.328358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.328312 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhd2\" (UniqueName: \"kubernetes.io/projected/2a226ada-8eab-4c72-b8f4-26e476e9dece-kube-api-access-9qhd2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:33.744220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.744185 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" Nov 03 19:04:33.744220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.744204 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh" event={"ID":"b2de5c44-10cd-46c3-8386-25ad4b135c5d","Type":"ContainerDied","Data":"3b4d5419d66aef98fb4ed381b8ff58f53dd3266fc34b33f2858759d0b1559df6"} Nov 03 19:04:33.744803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.744291 4948 scope.go:122] "RemoveContainer" containerID="ac68b77d12a7c8f8c05ee7bdf48309437c2ce9409029943ceb14c1a96e116c85" Nov 03 19:04:33.746095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.746074 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" Nov 03 19:04:33.746182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.746075 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c" event={"ID":"2a226ada-8eab-4c72-b8f4-26e476e9dece","Type":"ContainerDied","Data":"2bf7fcf27ed358f8765d39e5fe730611f653845ed2a979d832e2c5bc1ec99f42"} Nov 03 19:04:33.747627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.747608 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" Nov 03 19:04:33.747762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.747632 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4" event={"ID":"049a88a5-7b87-4324-9b8c-ea46d263756f","Type":"ContainerDied","Data":"a62b494cbc8f0cbe000ace008fd7f16c822b5c6ff1c59ad91e2a0b063b925b63"} Nov 03 19:04:33.747762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.747668 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a62b494cbc8f0cbe000ace008fd7f16c822b5c6ff1c59ad91e2a0b063b925b63" Nov 03 19:04:33.768879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.768854 4948 scope.go:122] "RemoveContainer" containerID="3f7f5995a87da8fd231741f1927a9cf884673cae09a00bda707c92e1af792145" Nov 03 19:04:33.917748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.917691 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c"] Nov 03 19:04:33.923667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:33.923639 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-30b92815-c97b7d6c5-gpg5c"] Nov 03 19:04:34.088801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:34.088715 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh"] Nov 03 19:04:34.095051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:34.095008 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-30b92815-7984849f75-2fvrh"] Nov 03 19:04:34.756579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:34.756544 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a226ada-8eab-4c72-b8f4-26e476e9dece" path="/var/lib/kubelet/pods/2a226ada-8eab-4c72-b8f4-26e476e9dece/volumes" Nov 03 19:04:34.757009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:34.756993 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2de5c44-10cd-46c3-8386-25ad4b135c5d" path="/var/lib/kubelet/pods/b2de5c44-10cd-46c3-8386-25ad4b135c5d/volumes" Nov 03 19:04:43.660249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:43.660208 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8"] Nov 03 19:04:43.708332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:43.708289 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90774e61-5qmb8"] Nov 03 19:04:44.756109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:44.756067 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0926fd17-9755-47fa-9784-c1189d6922a2" path="/var/lib/kubelet/pods/0926fd17-9755-47fa-9784-c1189d6922a2/volumes" Nov 03 19:04:45.649656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.649599 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn"] Nov 03 19:04:45.651121 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.651087 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2a226ada-8eab-4c72-b8f4-26e476e9dece" containerName="payload-pre-processing" Nov 03 19:04:45.651300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.651282 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a226ada-8eab-4c72-b8f4-26e476e9dece" containerName="payload-pre-processing" Nov 03 19:04:45.651418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.651405 4948 cpu_manager.go:414] "containerMap: removing container" podUID="049a88a5-7b87-4324-9b8c-ea46d263756f" containerName="revoke-keys" Nov 03 19:04:45.651512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.651499 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049a88a5-7b87-4324-9b8c-ea46d263756f" containerName="revoke-keys" Nov 03 19:04:45.651655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.651642 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b2de5c44-10cd-46c3-8386-25ad4b135c5d" containerName="payload-processing" Nov 03 19:04:45.651770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.651755 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2de5c44-10cd-46c3-8386-25ad4b135c5d" containerName="payload-processing" Nov 03 19:04:45.652087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.652067 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2de5c44-10cd-46c3-8386-25ad4b135c5d" containerName="payload-processing" Nov 03 19:04:45.652299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.652284 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a226ada-8eab-4c72-b8f4-26e476e9dece" containerName="payload-pre-processing" Nov 03 19:04:45.652404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.652393 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="049a88a5-7b87-4324-9b8c-ea46d263756f" containerName="revoke-keys" Nov 03 19:04:45.739430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.739263 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn"] Nov 03 19:04:45.740016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.739922 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.861344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.861310 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-envoy\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.861815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.861472 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfwg\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-kube-api-access-5mfwg\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.861815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.861518 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-token\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.861815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.861800 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-credential-socket\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.861960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.861843 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-podinfo\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.861960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.861929 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-certs\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.862044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.862029 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5837ad6-6036-4611-aa0d-149b7f52c356-istiod-ca-cert\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.862089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.862062 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-data\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.862145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.862129 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-socket\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.880789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.880761 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-82cd41ab-gw-openshift-default-dockercfg-lt552\"" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.967821 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-credential-socket\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.967866 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-podinfo\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.967935 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-certs\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.968043 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5837ad6-6036-4611-aa0d-149b7f52c356-istiod-ca-cert\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.968091 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-data\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.968150 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-socket\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.968205 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-envoy\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.968341 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfwg\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-kube-api-access-5mfwg\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.968396 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-token\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.969758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.969231 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-credential-socket\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.970610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.969943 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-data\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.970610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.970184 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-certs\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.972526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.970927 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5837ad6-6036-4611-aa0d-149b7f52c356-istiod-ca-cert\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:45.988730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:45.986424 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-socket\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:46.073489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.073410 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-token\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:46.091744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.088976 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfwg\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-kube-api-access-5mfwg\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:46.103724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.102587 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-podinfo\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:46.107582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.107551 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-envoy\") pod \"e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:46.139141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.139062 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:46.463185 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.463145 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j"] Nov 03 19:04:46.469014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.468989 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.523234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.523161 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9e277a-openshift-default-dockercfg-w7jdw\"" Nov 03 19:04:46.526262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.526082 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j"] Nov 03 19:04:46.580368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580330 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.580545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580474 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz45z\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-kube-api-access-vz45z\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.580601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580551 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-credential-socket\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.580651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580633 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-token\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.580692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580676 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-envoy\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.580809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580795 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-certs\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.580911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580894 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-podinfo\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.580948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580938 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-socket\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.581000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.580988 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-data\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.783463 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-socket\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.783565 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-data\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.783652 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.783784 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz45z\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-kube-api-access-vz45z\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.783893 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-credential-socket\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.783958 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-token\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.784023 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-envoy\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.784104 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-certs\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.784197 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-podinfo\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.787835 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-podinfo\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.788163 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-socket\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.788429 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-data\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.794674 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-credential-socket\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.794900 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.797914 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.798004 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:46.798774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.798046 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:46.801846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.800626 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-certs\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.824722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.809465 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-envoy\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.844453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.844336 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn"] Nov 03 19:04:46.857239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.857188 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-token\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:46.858569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:46.858546 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz45z\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-kube-api-access-vz45z\") pod \"e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:47.056708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.056618 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" event={"ID":"e5837ad6-6036-4611-aa0d-149b7f52c356","Type":"ContainerStarted","Data":"e90fffd5428ef2ccb60ad4799b452a6724554f3f4b4f69f2a951158d23cebcc0"} Nov 03 19:04:47.056708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.056672 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" event={"ID":"e5837ad6-6036-4611-aa0d-149b7f52c356","Type":"ContainerStarted","Data":"2646aa67025f1f61f7243a6b3b66d546795caf2f4dd7f7df1358913e20f31bf4"} Nov 03 19:04:47.085480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.085449 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:47.142751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.140171 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:47.142751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.141212 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:04:47.142751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.141271 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 19:04:47.189400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.189335 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podStartSLOduration=2.189315722 podStartE2EDuration="2.189315722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:47.184950642 +0000 UTC m=+1616.667946569" watchObservedRunningTime="2026-11-03 19:04:47.189315722 +0000 UTC m=+1616.672311636" Nov 03 19:04:47.564420 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:04:47.564390 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e0c00c_5a75_46d0_8042_d43c2c776cd8.slice/crio-848aaf2bc170f9e44b370ecdfef3a1442df01dde3f838376fbbba68526d4fea8 WatchSource:0}: Error finding container 848aaf2bc170f9e44b370ecdfef3a1442df01dde3f838376fbbba68526d4fea8: Status 404 returned error can't find the container with id 848aaf2bc170f9e44b370ecdfef3a1442df01dde3f838376fbbba68526d4fea8 Nov 03 19:04:47.566762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.566689 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j"] Nov 03 19:04:47.567570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.567540 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:47.567625 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.567603 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:47.567660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:47.567632 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:04:48.072831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:48.072783 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" event={"ID":"23e0c00c-5a75-46d0-8042-d43c2c776cd8","Type":"ContainerStarted","Data":"cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37"} Nov 03 19:04:48.073335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:48.072893 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" event={"ID":"23e0c00c-5a75-46d0-8042-d43c2c776cd8","Type":"ContainerStarted","Data":"848aaf2bc170f9e44b370ecdfef3a1442df01dde3f838376fbbba68526d4fea8"} Nov 03 19:04:48.085845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:48.085815 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:48.090945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:48.090899 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:48.144748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:48.144693 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:48.206198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:48.206143 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podStartSLOduration=2.20612908 podStartE2EDuration="2.20612908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:48.197465914 +0000 UTC m=+1617.680461850" watchObservedRunningTime="2026-11-03 19:04:48.20612908 +0000 UTC m=+1617.689124994" Nov 03 19:04:49.085886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:49.085116 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:49.085886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:49.085187 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:49.088745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:49.086938 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:04:49.088745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:49.087000 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:04:52.702741 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.702019 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4"] Nov 03 19:04:52.733727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.732781 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:52.825075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.824303 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pq9\" (UniqueName: \"kubernetes.io/projected/c3c8e353-f35c-4d81-8a3a-2103adcacb91-kube-api-access-c9pq9\") pod \"payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:52.825075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.824395 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c8e353-f35c-4d81-8a3a-2103adcacb91-plugins-config-volume\") pod \"payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:52.838718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.835978 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-82cd41ab\"" Nov 03 19:04:52.898121 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.898090 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-82cd41ab-dockercfg-crplt\"" Nov 03 19:04:52.955353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.930245 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pq9\" (UniqueName: \"kubernetes.io/projected/c3c8e353-f35c-4d81-8a3a-2103adcacb91-kube-api-access-c9pq9\") pod \"payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:52.955353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.930365 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c8e353-f35c-4d81-8a3a-2103adcacb91-plugins-config-volume\") pod \"payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:52.955353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:52.931860 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c8e353-f35c-4d81-8a3a-2103adcacb91-plugins-config-volume\") pod \"payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:53.263774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:53.257539 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4"] Nov 03 19:04:53.290969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:53.290896 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pq9\" (UniqueName: \"kubernetes.io/projected/c3c8e353-f35c-4d81-8a3a-2103adcacb91-kube-api-access-c9pq9\") pod \"payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:53.520865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:53.520789 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:53.967679 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:53.966009 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4"] Nov 03 19:04:53.970474 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:04:53.970430 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c8e353_f35c_4d81_8a3a_2103adcacb91.slice/crio-a77462f22e853935bfc645aa1028eb8f1c42d190f44e274b0eaecada1465bbc1 WatchSource:0}: Error finding container a77462f22e853935bfc645aa1028eb8f1c42d190f44e274b0eaecada1465bbc1: Status 404 returned error can't find the container with id a77462f22e853935bfc645aa1028eb8f1c42d190f44e274b0eaecada1465bbc1 Nov 03 19:04:54.403074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:54.403029 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" event={"ID":"c3c8e353-f35c-4d81-8a3a-2103adcacb91","Type":"ContainerStarted","Data":"a77462f22e853935bfc645aa1028eb8f1c42d190f44e274b0eaecada1465bbc1"} Nov 03 19:04:55.709567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:55.709525 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" event={"ID":"c3c8e353-f35c-4d81-8a3a-2103adcacb91","Type":"ContainerStarted","Data":"750dc569c6114292f73d1a2e2d2a20f0cef814676d2857aac565bebb5aa35f8b"} Nov 03 19:04:55.710145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:55.709821 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:04:55.835593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:55.835540 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" podStartSLOduration=3.8355250119999997 podStartE2EDuration="3.835525012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:55.828606315 +0000 UTC m=+1625.311602231" watchObservedRunningTime="2026-11-03 19:04:55.835525012 +0000 UTC m=+1625.318520925" Nov 03 19:04:56.140879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:56.140779 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t"] Nov 03 19:04:56.185719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:56.171517 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f810b24-tv79t"] Nov 03 19:04:56.476940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:56.476902 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6"] Nov 03 19:04:56.482690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:56.482664 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f198bfb-xbpf6"] Nov 03 19:04:56.756029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:56.755955 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c" path="/var/lib/kubelet/pods/6bfe0a7d-90c1-4cf2-ae81-29ca81f0bd6c/volumes" Nov 03 19:04:56.756476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:56.756460 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcfc66a-658f-43f5-99b2-2600b76d3806" path="/var/lib/kubelet/pods/cbcfc66a-658f-43f5-99b2-2600b76d3806/volumes" Nov 03 19:04:57.551082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:57.551050 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27"] Nov 03 19:04:57.606068 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:57.606033 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-30b92815-l6k27"] Nov 03 19:04:58.171986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:58.171947 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz"] Nov 03 19:04:58.183978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:58.183920 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" containerID="cri-o://a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d" gracePeriod=30 Nov 03 19:04:58.816719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:04:58.810973 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a09d5f-8abd-4d07-a780-a01e7bc17c39" path="/var/lib/kubelet/pods/c4a09d5f-8abd-4d07-a780-a01e7bc17c39/volumes" Nov 03 19:05:02.843080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:02.843017 4948 patch_prober.go:29] interesting pod/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:05:02.844002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:02.843942 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:05:02.902319 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:05:02.899822 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:05:03.899777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.899746 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:05:03.904124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.904093 4948 generic.go:356] "Generic (PLEG): container finished" podID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerID="a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d" exitCode=0 Nov 03 19:05:03.904247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.904146 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" Nov 03 19:05:03.904247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.904218 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" event={"ID":"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a","Type":"ContainerDied","Data":"a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d"} Nov 03 19:05:03.904347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.904284 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz" event={"ID":"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a","Type":"ContainerDied","Data":"838e0e57a20bb89218aa4e864a6b469c4de9979547bbe73afe47c09836d2221e"} Nov 03 19:05:03.904347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.904317 4948 scope.go:122] "RemoveContainer" containerID="a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d" Nov 03 19:05:03.916342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.916320 4948 scope.go:122] "RemoveContainer" containerID="dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350" Nov 03 19:05:03.927656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.927634 4948 scope.go:122] "RemoveContainer" containerID="a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d" Nov 03 19:05:03.928021 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:05:03.927997 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d\": container with ID starting with a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d not found: ID does not exist" containerID="a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d" Nov 03 19:05:03.928080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.928030 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d"} err="failed to get container status \"a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d\": rpc error: code = NotFound desc = could not find container \"a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d\": container with ID starting with a258c3d5f5b1085b2844f202f3a230702488e90507d170a09ebf958afb7f607d not found: ID does not exist" Nov 03 19:05:03.928080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.928048 4948 scope.go:122] "RemoveContainer" containerID="dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350" Nov 03 19:05:03.928364 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:05:03.928341 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350\": container with ID starting with dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350 not found: ID does not exist" containerID="dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350" Nov 03 19:05:03.928432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.928376 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350"} err="failed to get container status \"dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350\": rpc error: code = NotFound desc = could not find container \"dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350\": container with ID starting with dd5a1bbcdf56402d6253831fa9dae7918f1344d73fcc5ca0d85c9998f31dc350 not found: ID does not exist" Nov 03 19:05:03.959016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.958987 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-socket\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959150 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istiod-ca-cert\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959200 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-data\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959227 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-envoy\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959308 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-token\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-token\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959399 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-certs\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959432 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-credential-socket\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959498 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-podinfo\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959309 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-socket" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:05:03.959645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959503 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-data" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:05:03.959645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959546 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-kube-api-access-brsx9\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-kube-api-access-brsx9\") pod \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\" (UID: \"f002b4ad-77d9-4c88-b2bd-d49102a8ef7a\") " Nov 03 19:05:03.959645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959573 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istiod-ca-cert" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:05:03.959832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959772 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-credential-socket" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:05:03.959832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.959806 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-certs" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:05:03.960594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.960494 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:03.960594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.960524 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:03.960594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.960539 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:03.960594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.960554 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:03.960594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.960568 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:03.961633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.961611 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-envoy" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:05:03.961982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.961960 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-podinfo" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:05:03.962103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.962083 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-token" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:05:03.962151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:03.962115 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-kube-api-access-brsx9" pod "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" (UID: "f002b4ad-77d9-4c88-b2bd-d49102a8ef7a"). InnerVolumeSpecName "kube-api-access-brsx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:05:04.062435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:04.062401 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:04.062435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:04.062431 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:04.062435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:04.062441 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:04.062808 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:04.062450 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brsx9\" (UniqueName: \"kubernetes.io/projected/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a-kube-api-access-brsx9\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:05:04.334253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:04.334169 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz"] Nov 03 19:05:04.339957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:04.339930 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6f198bfb-openshift-default-b76b5d689-dgcgz"] Nov 03 19:05:04.756980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:04.756943 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" path="/var/lib/kubelet/pods/f002b4ad-77d9-4c88-b2bd-d49102a8ef7a/volumes" Nov 03 19:05:06.815885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.813345 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:05:06.861764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.860365 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8"] Nov 03 19:05:06.861764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.861599 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" Nov 03 19:05:06.861764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.861624 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" Nov 03 19:05:06.861764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.861655 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" Nov 03 19:05:06.861764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.861664 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" Nov 03 19:05:06.865949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.862440 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" Nov 03 19:05:06.865949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.862478 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f002b4ad-77d9-4c88-b2bd-d49102a8ef7a" containerName="istio-proxy" Nov 03 19:05:06.870906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.870844 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:06.885863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.873626 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8"] Nov 03 19:05:06.946394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.946359 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6765e083-dockercfg-x9nt5\"" Nov 03 19:05:06.946880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:06.946853 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6765e083\"" Nov 03 19:05:07.126762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.103329 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks49h\" (UniqueName: \"kubernetes.io/projected/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-kube-api-access-ks49h\") pod \"payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:07.126762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.103735 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-plugins-config-volume\") pod \"payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:07.196734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.196262 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz"] Nov 03 19:05:07.213492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.213434 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-plugins-config-volume\") pod \"payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:07.213755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.213733 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks49h\" (UniqueName: \"kubernetes.io/projected/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-kube-api-access-ks49h\") pod \"payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:07.215304 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.215267 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-plugins-config-volume\") pod \"payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:07.321325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.321290 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz"] Nov 03 19:05:07.322362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.322302 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.354146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.354041 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks49h\" (UniqueName: \"kubernetes.io/projected/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-kube-api-access-ks49h\") pod \"payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:07.435237 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.435197 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc897\" (UniqueName: \"kubernetes.io/projected/e3097574-7365-4487-9838-fe8fc220385b-kube-api-access-dc897\") pod \"payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.435430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.435412 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3097574-7365-4487-9838-fe8fc220385b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.466993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.466962 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:07.537831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.537804 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc897\" (UniqueName: \"kubernetes.io/projected/e3097574-7365-4487-9838-fe8fc220385b-kube-api-access-dc897\") pod \"payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.537994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.537982 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3097574-7365-4487-9838-fe8fc220385b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.542794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.539244 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3097574-7365-4487-9838-fe8fc220385b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.602402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.602312 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc897\" (UniqueName: \"kubernetes.io/projected/e3097574-7365-4487-9838-fe8fc220385b-kube-api-access-dc897\") pod \"payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.678675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.678585 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:07.854904 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:05:07.854872 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4375cc8_9a99_4fd7_a0d3_fead6d78cfa8.slice/crio-775d85766c2ffdc61165f2e33b56c776d1b29d5613c2332cbaf8f211943335fa WatchSource:0}: Error finding container 775d85766c2ffdc61165f2e33b56c776d1b29d5613c2332cbaf8f211943335fa: Status 404 returned error can't find the container with id 775d85766c2ffdc61165f2e33b56c776d1b29d5613c2332cbaf8f211943335fa Nov 03 19:05:07.857663 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:07.857639 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8"] Nov 03 19:05:08.069028 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:05:08.068987 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3097574_7365_4487_9838_fe8fc220385b.slice/crio-ca3c164b73e8ba286fad938f0ff80bca727a8924d8fb374a642dc2bb2ab91de5 WatchSource:0}: Error finding container ca3c164b73e8ba286fad938f0ff80bca727a8924d8fb374a642dc2bb2ab91de5: Status 404 returned error can't find the container with id ca3c164b73e8ba286fad938f0ff80bca727a8924d8fb374a642dc2bb2ab91de5 Nov 03 19:05:08.070680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.070654 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz"] Nov 03 19:05:08.382078 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.382044 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" event={"ID":"e3097574-7365-4487-9838-fe8fc220385b","Type":"ContainerStarted","Data":"95ec0a01cb7bdbb996c0e36c456ebeea33b7ba5157e7eb3d72d7cceac9e8ed19"} Nov 03 19:05:08.382289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.382102 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" event={"ID":"e3097574-7365-4487-9838-fe8fc220385b","Type":"ContainerStarted","Data":"ca3c164b73e8ba286fad938f0ff80bca727a8924d8fb374a642dc2bb2ab91de5"} Nov 03 19:05:08.382289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.382136 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:08.384488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.384455 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" event={"ID":"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8","Type":"ContainerStarted","Data":"7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55"} Nov 03 19:05:08.384637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.384510 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" event={"ID":"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8","Type":"ContainerStarted","Data":"775d85766c2ffdc61165f2e33b56c776d1b29d5613c2332cbaf8f211943335fa"} Nov 03 19:05:08.384637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.384549 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:08.504447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.504355 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" podStartSLOduration=2.5043387040000002 podStartE2EDuration="2.504338704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:08.499895319 +0000 UTC m=+1637.982891234" watchObservedRunningTime="2026-11-03 19:05:08.504338704 +0000 UTC m=+1637.987334618" Nov 03 19:05:08.656771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:08.649905 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" podStartSLOduration=2.649883874 podStartE2EDuration="2.649883874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:08.64291426 +0000 UTC m=+1638.125910177" watchObservedRunningTime="2026-11-03 19:05:08.649883874 +0000 UTC m=+1638.132879789" Nov 03 19:05:12.054140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:12.054092 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t"] Nov 03 19:05:12.104434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:12.104397 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e23c0d-m492t"] Nov 03 19:05:12.755426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:12.755389 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fcf34c-d285-4be6-ac9b-be88bc4f457f" path="/var/lib/kubelet/pods/08fcf34c-d285-4be6-ac9b-be88bc4f457f/volumes" Nov 03 19:05:19.399217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:19.399184 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:05:19.399833 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:19.399558 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:05:32.531166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.531089 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm"] Nov 03 19:05:32.537161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.537140 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.631864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.631820 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632231 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632386 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj48s\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-kube-api-access-rj48s\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632422 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632605 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632672 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632804 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632741 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632923 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632896 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.632972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.632949 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.652319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.652288 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm"] Nov 03 19:05:32.734928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.734898 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.734937 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.734978 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.735036 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.735126 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj48s\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-kube-api-access-rj48s\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.735156 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.735317 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.735404 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.735459 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.735718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.735647 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.736118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.736093 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.736207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.736190 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.736246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.736222 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.736457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.736440 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.737394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.737373 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.737764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.737742 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.797687 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:05:32.797603 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:05:32.797945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.797924 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.798385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.798357 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj48s\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-kube-api-access-rj48s\") pod \"maas-default-gateway-openshift-default-7854949c9-r6cjm\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:32.853966 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:32.853931 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:33.301089 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:05:33.301048 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19168cf_c7fb_4d33_a35d_32c3ea6cc372.slice/crio-b451afb21ad14ec0f49c97106bcd3c8a9d521583d6eeaeeb80dcf93ea15b14c8 WatchSource:0}: Error finding container b451afb21ad14ec0f49c97106bcd3c8a9d521583d6eeaeeb80dcf93ea15b14c8: Status 404 returned error can't find the container with id b451afb21ad14ec0f49c97106bcd3c8a9d521583d6eeaeeb80dcf93ea15b14c8 Nov 03 19:05:33.303488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.303464 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm"] Nov 03 19:05:33.304576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.304532 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:33.304681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.304617 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:33.304681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.304655 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:33.709400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.709361 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" event={"ID":"e19168cf-c7fb-4d33-a35d-32c3ea6cc372","Type":"ContainerStarted","Data":"0eaedeb657271febffca14817fdf9134339da1f3a407a2e77f5db97a5f76b2af"} Nov 03 19:05:33.709931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.709427 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" event={"ID":"e19168cf-c7fb-4d33-a35d-32c3ea6cc372","Type":"ContainerStarted","Data":"b451afb21ad14ec0f49c97106bcd3c8a9d521583d6eeaeeb80dcf93ea15b14c8"} Nov 03 19:05:33.836998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.836944 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podStartSLOduration=1.83692841 podStartE2EDuration="1.83692841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:33.834748114 +0000 UTC m=+1663.317744030" watchObservedRunningTime="2026-11-03 19:05:33.83692841 +0000 UTC m=+1663.319924323" Nov 03 19:05:33.854642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:33.854616 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:34.874724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:34.862100 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 19:05:34.874724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:34.862168 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 19:05:35.876722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:35.863115 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 19:05:35.876722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:35.863197 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 19:05:36.073448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:36.073405 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts"] Nov 03 19:05:36.093606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:36.093570 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ed7bc1-cnjts"] Nov 03 19:05:36.781762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:36.781729 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7f2364-589b-49da-8ef1-7df46cbe4a92" path="/var/lib/kubelet/pods/bc7f2364-589b-49da-8ef1-7df46cbe4a92/volumes" Nov 03 19:05:36.859732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:36.858041 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 19:05:36.859732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:36.858113 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 19:05:37.871722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:37.858061 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 19:05:37.871722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:37.858151 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 19:05:38.870122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:38.870083 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 19:05:38.870534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:38.870503 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 19:05:39.855053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:39.855010 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 19:05:39.855462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:39.855074 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 19:05:40.856226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:40.855743 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 19:05:40.856226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:40.855823 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 19:05:42.291996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:42.291968 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:42.292518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:42.292489 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:42.293828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:42.293805 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:05:47.664843 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.664807 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln"] Nov 03 19:05:47.669922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.669897 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766193 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766239 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f18b1dc-f307-4945-800f-0af6509176f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766344 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpr96\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-kube-api-access-tpr96\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766487 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766542 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766569 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766730 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f18b1dc-f307-4945-800f-0af6509176f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766768 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.766946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.766798 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.796380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.796341 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln"] Nov 03 19:05:47.868053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868006 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f18b1dc-f307-4945-800f-0af6509176f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868127 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868165 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868224 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868274 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f18b1dc-f307-4945-800f-0af6509176f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868411 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpr96\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-kube-api-access-tpr96\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868598 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868650 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868673 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868769 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868802 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.868952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.868883 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f18b1dc-f307-4945-800f-0af6509176f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.869517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.869489 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.872748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.869982 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.872748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.872212 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.872748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.872431 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f18b1dc-f307-4945-800f-0af6509176f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.933271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.933246 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpr96\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-kube-api-access-tpr96\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.933439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.933414 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-c62ln\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:47.990981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:47.989474 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:48.439166 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:05:48.439130 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f18b1dc_f307_4945_800f_0af6509176f5.slice/crio-05bfe58fd64b8a978093908523fc492642be51babcfc982ff707d0310df28f33 WatchSource:0}: Error finding container 05bfe58fd64b8a978093908523fc492642be51babcfc982ff707d0310df28f33: Status 404 returned error can't find the container with id 05bfe58fd64b8a978093908523fc492642be51babcfc982ff707d0310df28f33 Nov 03 19:05:48.441187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:48.441159 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln"] Nov 03 19:05:48.442140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:48.442104 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:48.442244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:48.442175 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:48.442244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:48.442206 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:48.943391 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:48.943351 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" event={"ID":"9f18b1dc-f307-4945-800f-0af6509176f5","Type":"ContainerStarted","Data":"1ea0376aa361e4de421157bfc176ab31e6d49442cdea9f3f47366b8f2053aa7f"} Nov 03 19:05:48.943868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:48.943413 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" event={"ID":"9f18b1dc-f307-4945-800f-0af6509176f5","Type":"ContainerStarted","Data":"05bfe58fd64b8a978093908523fc492642be51babcfc982ff707d0310df28f33"} Nov 03 19:05:48.993718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:48.990861 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:49.077128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:49.077055 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podStartSLOduration=2.077029618 podStartE2EDuration="2.077029618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:49.070253973 +0000 UTC m=+1678.553249900" watchObservedRunningTime="2026-11-03 19:05:49.077029618 +0000 UTC m=+1678.560025533" Nov 03 19:05:49.992463 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:49.992426 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:05:49.992901 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:49.992483 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:05:50.990528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:50.990471 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:05:50.990797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:50.990553 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:05:51.994097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:51.991380 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:05:51.994097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:51.991460 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:05:52.997911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:52.994873 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:05:52.997911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:52.994974 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:05:53.116754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.115233 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59"] Nov 03 19:05:53.128613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.128579 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.130315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.130285 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59"] Nov 03 19:05:53.186753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.186714 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-770e67-openshift-default-dockercfg-2zx6s\"" Nov 03 19:05:53.222828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.222792 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istiod-ca-cert\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.222994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.222856 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-socket\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.222994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.222946 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-certs\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.223121 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.223105 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-envoy\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.223156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.223130 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-data\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.223191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.223165 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-podinfo\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.223303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.223286 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-token\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.223348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.223337 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-credential-socket\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.223457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.223444 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44mn8\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-kube-api-access-44mn8\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.342774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.327991 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-credential-socket\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.348721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.343134 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44mn8\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-kube-api-access-44mn8\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.369723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.349062 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istiod-ca-cert\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.370142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.334414 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-credential-socket\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.370288 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.365169 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istiod-ca-cert\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.370469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.370434 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-socket\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.370649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.370621 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-certs\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.371041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.371023 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-envoy\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.371188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.371174 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-data\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.371380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.371364 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-podinfo\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.371620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.371602 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-token\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.383788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.377097 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-certs\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.384631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.384590 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-socket\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.385204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.385167 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-data\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.524983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.493993 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44mn8\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-kube-api-access-44mn8\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.524983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.494012 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-envoy\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.524983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.494672 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-podinfo\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.536720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.535287 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-token\") pod \"e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.748294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.748258 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:53.990926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.990890 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:05:53.991104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:53.990945 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:05:54.224085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:54.224059 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59"] Nov 03 19:05:54.224497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:54.224465 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:54.224567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:54.224551 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:54.224625 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:54.224598 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:05:54.990859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:54.990827 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:05:54.991007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:54.990884 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:05:55.058819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.058781 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9"] Nov 03 19:05:55.111228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.111193 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380f942e-gqxs9"] Nov 03 19:05:55.137956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.137922 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" event={"ID":"ac010993-8cde-4557-a9fe-ec54ab8dc11b","Type":"ContainerStarted","Data":"17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95"} Nov 03 19:05:55.138132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.137976 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" event={"ID":"ac010993-8cde-4557-a9fe-ec54ab8dc11b","Type":"ContainerStarted","Data":"3f0c3d52fb843b2d486e713db9b956d71eae6386f87f4f189ea181821599c312"} Nov 03 19:05:55.264193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.264086 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podStartSLOduration=3.264071125 podStartE2EDuration="3.264071125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:55.260602416 +0000 UTC m=+1684.743598330" watchObservedRunningTime="2026-11-03 19:05:55.264071125 +0000 UTC m=+1684.747067038" Nov 03 19:05:55.748933 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.748895 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:55.756953 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.756926 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:55.990863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.990821 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:05:55.991040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:55.990896 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:05:56.155914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:56.155809 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:56.157221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:56.157192 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:05:56.756576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:56.756533 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0851f0a5-62e9-479b-a7b2-7ae682b23ca2" path="/var/lib/kubelet/pods/0851f0a5-62e9-479b-a7b2-7ae682b23ca2/volumes" Nov 03 19:05:57.603262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:57.603226 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:57.604268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:57.604237 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:57.605117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:57.605091 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:05:58.053860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:58.053812 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb"] Nov 03 19:05:58.060175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:58.060142 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621c6a45-6cpzb"] Nov 03 19:05:58.281559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:58.281525 4948 scope.go:122] "RemoveContainer" containerID="a5ca5f73f5a48d9945176e7f5087e73dbfb96d4cd9838d88e746b14870aaa58b" Nov 03 19:05:58.293262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:58.293242 4948 scope.go:122] "RemoveContainer" containerID="1382db2ade5075e7dea829ca3e62763879913af7e323c0b7f6d09d39344843bb" Nov 03 19:05:58.756929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:05:58.756872 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb86547-af7a-419e-b2e0-8515a3ef3730" path="/var/lib/kubelet/pods/ccb86547-af7a-419e-b2e0-8515a3ef3730/volumes" Nov 03 19:06:01.057489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:01.057450 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx"] Nov 03 19:06:01.106666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:01.106631 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f1e9946-b5shx"] Nov 03 19:06:01.871809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:01.871762 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq"] Nov 03 19:06:01.889272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:01.887063 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:01.952672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:01.952636 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-770e67\"" Nov 03 19:06:02.396495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.396461 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq"] Nov 03 19:06:02.415732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.415660 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:02.416050 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.416012 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-kube-api-access-z72bb\") pod \"payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:02.533764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.490163 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v"] Nov 03 19:06:02.561552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.561508 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-kube-api-access-z72bb\") pod \"payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:02.561867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.561843 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:02.563760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.563563 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:02.582430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.581801 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:02.582430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.581871 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v"] Nov 03 19:06:02.627045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.627011 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-kube-api-access-z72bb\") pod \"payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:02.636491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.636469 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-770e67-dockercfg-mrp7x\"" Nov 03 19:06:02.756525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.756483 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6668c8b-1fe7-441a-a599-f748e448e518" path="/var/lib/kubelet/pods/c6668c8b-1fe7-441a-a599-f748e448e518/volumes" Nov 03 19:06:02.765352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.765324 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74868d4d-edf2-480a-b7b9-4aa75b7c667d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:02.765536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.765500 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdn9\" (UniqueName: \"kubernetes.io/projected/74868d4d-edf2-480a-b7b9-4aa75b7c667d-kube-api-access-2fdn9\") pod \"payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:02.796657 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:06:02.796612 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:06:02.867117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.867080 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74868d4d-edf2-480a-b7b9-4aa75b7c667d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:02.867313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.867206 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdn9\" (UniqueName: \"kubernetes.io/projected/74868d4d-edf2-480a-b7b9-4aa75b7c667d-kube-api-access-2fdn9\") pod \"payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:02.868170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.868140 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74868d4d-edf2-480a-b7b9-4aa75b7c667d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:02.899758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.899718 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:02.930615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:02.930592 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdn9\" (UniqueName: \"kubernetes.io/projected/74868d4d-edf2-480a-b7b9-4aa75b7c667d-kube-api-access-2fdn9\") pod \"payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:03.214036 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:03.214000 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:03.301356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:03.301322 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq"] Nov 03 19:06:03.314899 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:06:03.314861 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5471ae_666f_44b6_a472_1cb21c9c4e0a.slice/crio-7345077d4e19be237202253263a0dc44394e729d7b3156a3c14079bd7b0c04e0 WatchSource:0}: Error finding container 7345077d4e19be237202253263a0dc44394e729d7b3156a3c14079bd7b0c04e0: Status 404 returned error can't find the container with id 7345077d4e19be237202253263a0dc44394e729d7b3156a3c14079bd7b0c04e0 Nov 03 19:06:03.577243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:03.577157 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" event={"ID":"fe5471ae-666f-44b6-a472-1cb21c9c4e0a","Type":"ContainerStarted","Data":"d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c"} Nov 03 19:06:03.577243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:03.577228 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" event={"ID":"fe5471ae-666f-44b6-a472-1cb21c9c4e0a","Type":"ContainerStarted","Data":"7345077d4e19be237202253263a0dc44394e729d7b3156a3c14079bd7b0c04e0"} Nov 03 19:06:03.577746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:03.577257 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:03.608109 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:06:03.608079 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74868d4d_edf2_480a_b7b9_4aa75b7c667d.slice/crio-4a7c626b4862ac1a27051d34acc84b565ff98fd7bb55471d794b232caf026a11 WatchSource:0}: Error finding container 4a7c626b4862ac1a27051d34acc84b565ff98fd7bb55471d794b232caf026a11: Status 404 returned error can't find the container with id 4a7c626b4862ac1a27051d34acc84b565ff98fd7bb55471d794b232caf026a11 Nov 03 19:06:03.610317 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:03.610294 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v"] Nov 03 19:06:03.700012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:03.699957 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" podStartSLOduration=2.699941882 podStartE2EDuration="2.699941882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:03.695278312 +0000 UTC m=+1693.178274225" watchObservedRunningTime="2026-11-03 19:06:03.699941882 +0000 UTC m=+1693.182937795" Nov 03 19:06:04.590269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:04.590217 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" event={"ID":"74868d4d-edf2-480a-b7b9-4aa75b7c667d","Type":"ContainerStarted","Data":"5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9"} Nov 03 19:06:04.590800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:04.590291 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" event={"ID":"74868d4d-edf2-480a-b7b9-4aa75b7c667d","Type":"ContainerStarted","Data":"4a7c626b4862ac1a27051d34acc84b565ff98fd7bb55471d794b232caf026a11"} Nov 03 19:06:04.590800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:04.590327 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:04.714570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:04.714512 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" podStartSLOduration=3.714497109 podStartE2EDuration="3.714497109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:04.708649697 +0000 UTC m=+1694.191645610" watchObservedRunningTime="2026-11-03 19:06:04.714497109 +0000 UTC m=+1694.197493022" Nov 03 19:06:08.064722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:08.063035 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk"] Nov 03 19:06:08.146724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:08.146289 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b3898a-n6khk"] Nov 03 19:06:09.118719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:09.110745 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047fa32c-c3e4-4fde-b5ea-37baaf608c3d" path="/var/lib/kubelet/pods/047fa32c-c3e4-4fde-b5ea-37baaf608c3d/volumes" Nov 03 19:06:10.902783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:10.902666 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6"] Nov 03 19:06:10.925536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:10.925504 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:10.991869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:10.991837 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5ac14ff6-openshift-default-dockercfg-pqd9r\"" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.080473 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-token\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.080781 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-credential-socket\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.080854 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-certs\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.080914 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnkn\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-kube-api-access-9nnkn\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.081014 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-data\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.081049 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istiod-ca-cert\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.081077 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-socket\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.081133 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-envoy\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.095724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.081250 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-podinfo\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211120 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-credential-socket\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211212 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-certs\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211270 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnkn\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-kube-api-access-9nnkn\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211361 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-data\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211394 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istiod-ca-cert\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211423 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-socket\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211474 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-envoy\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211578 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-podinfo\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.211635 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-token\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.228320 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-data\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.228606 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-credential-socket\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.228880 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-certs\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.229464 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-socket\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.230164 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istiod-ca-cert\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.403924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.403890 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-podinfo\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.433725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.418371 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-envoy\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.464194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.453360 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-token\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.464194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.453416 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6"] Nov 03 19:06:11.538349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.538210 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnkn\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-kube-api-access-9nnkn\") pod \"e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:11.739388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:11.739356 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:12.224074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:12.224032 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6"] Nov 03 19:06:12.285866 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:06:12.285834 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9a4423_52ad_4f7b_b92e_49d008755cbe.slice/crio-9dcd2011835ef9a6feab33b9c3db03ccc192d3672ec26c4f657fb9bd3818f3aa WatchSource:0}: Error finding container 9dcd2011835ef9a6feab33b9c3db03ccc192d3672ec26c4f657fb9bd3818f3aa: Status 404 returned error can't find the container with id 9dcd2011835ef9a6feab33b9c3db03ccc192d3672ec26c4f657fb9bd3818f3aa Nov 03 19:06:12.288665 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:12.288628 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:12.288790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:12.288720 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:12.288790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:12.288757 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:13.087370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:13.087331 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" event={"ID":"9a9a4423-52ad-4f7b-b92e-49d008755cbe","Type":"ContainerStarted","Data":"08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d"} Nov 03 19:06:13.087583 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:13.087392 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" event={"ID":"9a9a4423-52ad-4f7b-b92e-49d008755cbe","Type":"ContainerStarted","Data":"9dcd2011835ef9a6feab33b9c3db03ccc192d3672ec26c4f657fb9bd3818f3aa"} Nov 03 19:06:13.287523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:13.287465 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podStartSLOduration=3.287450523 podStartE2EDuration="3.287450523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:13.282037191 +0000 UTC m=+1702.765033118" watchObservedRunningTime="2026-11-03 19:06:13.287450523 +0000 UTC m=+1702.770446498" Nov 03 19:06:13.739834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:13.739795 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:13.744891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:13.744863 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:14.105316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:14.104414 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:14.105491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:14.105402 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:06:14.595119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:14.595084 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:06:15.076724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:15.075746 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb"] Nov 03 19:06:15.096486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:15.096452 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81a9bd8-zz8nb"] Nov 03 19:06:15.640503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:15.640473 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:06:16.758758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:16.758720 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3398b2e-0c7a-4261-8152-b730c6124acd" path="/var/lib/kubelet/pods/a3398b2e-0c7a-4261-8152-b730c6124acd/volumes" Nov 03 19:06:18.130827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.129916 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b"] Nov 03 19:06:18.146415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.146389 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.192628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.191597 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b"] Nov 03 19:06:18.202728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.199991 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:06:18.216949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.216918 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkqb\" (UniqueName: \"kubernetes.io/projected/f475bdd3-4f45-4fdc-a884-0580f9450c75-kube-api-access-plkqb\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.217110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.216984 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.217392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.217360 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.234188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.234164 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:06:18.234528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.234499 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:06:18.235117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.234782 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:06:18.318580 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.318542 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.318820 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.318799 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.318892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.318877 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkqb\" (UniqueName: \"kubernetes.io/projected/f475bdd3-4f45-4fdc-a884-0580f9450c75-kube-api-access-plkqb\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.319442 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.319419 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.321066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.321046 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.381678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.381603 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkqb\" (UniqueName: \"kubernetes.io/projected/f475bdd3-4f45-4fdc-a884-0580f9450c75-kube-api-access-plkqb\") pod \"maas-api-revoke-sub-75130f0c-fcb4b\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.467604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.467560 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:18.885168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:18.885136 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b"] Nov 03 19:06:19.171011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:19.170926 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" event={"ID":"f475bdd3-4f45-4fdc-a884-0580f9450c75","Type":"ContainerStarted","Data":"cbeea112e5dbdb5fcf398b7b490d778e4a51b82beb23343fd862d760a872aafe"} Nov 03 19:06:19.475864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:19.474114 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": read tcp 10.132.0.2:51874->10.132.0.170:15021: read: connection reset by peer" start-of-body= Nov 03 19:06:19.475864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:19.474175 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": read tcp 10.132.0.2:51874->10.132.0.170:15021: read: connection reset by peer" Nov 03 19:06:20.338777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.338185 4948 generic.go:356] "Generic (PLEG): container finished" podID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerID="e90fffd5428ef2ccb60ad4799b452a6724554f3f4b4f69f2a951158d23cebcc0" exitCode=0 Nov 03 19:06:20.338777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.338613 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" event={"ID":"e5837ad6-6036-4611-aa0d-149b7f52c356","Type":"ContainerDied","Data":"e90fffd5428ef2ccb60ad4799b452a6724554f3f4b4f69f2a951158d23cebcc0"} Nov 03 19:06:20.339333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.339206 4948 scope.go:122] "RemoveContainer" containerID="e90fffd5428ef2ccb60ad4799b452a6724554f3f4b4f69f2a951158d23cebcc0" Nov 03 19:06:20.391931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.391883 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:20.392177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.392151 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:20.392267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.392216 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:20.487724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.484179 4948 generic.go:356] "Generic (PLEG): container finished" podID="f475bdd3-4f45-4fdc-a884-0580f9450c75" containerID="c337bed8b388b86fd6d777ffa2a9cd091e3ffc30a8b116b31588695e568e35b1" exitCode=0 Nov 03 19:06:20.487724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.484260 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" event={"ID":"f475bdd3-4f45-4fdc-a884-0580f9450c75","Type":"ContainerDied","Data":"c337bed8b388b86fd6d777ffa2a9cd091e3ffc30a8b116b31588695e568e35b1"} Nov 03 19:06:20.600773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.591530 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9"] Nov 03 19:06:20.649913 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.643725 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:20.743720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.734268 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5ac14ff6-dockercfg-jmjb2\"" Nov 03 19:06:20.743720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.734372 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5ac14ff6\"" Nov 03 19:06:20.888742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.886048 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc02de7-a8a0-4c12-b809-11034c3ca45e-plugins-config-volume\") pod \"payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:20.888742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:20.886508 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74b98\" (UniqueName: \"kubernetes.io/projected/ccc02de7-a8a0-4c12-b809-11034c3ca45e-kube-api-access-74b98\") pod \"payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:21.061234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.042292 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc02de7-a8a0-4c12-b809-11034c3ca45e-plugins-config-volume\") pod \"payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:21.061234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.042581 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74b98\" (UniqueName: \"kubernetes.io/projected/ccc02de7-a8a0-4c12-b809-11034c3ca45e-kube-api-access-74b98\") pod \"payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:21.061234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.049552 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9"] Nov 03 19:06:21.091731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.086268 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc02de7-a8a0-4c12-b809-11034c3ca45e-plugins-config-volume\") pod \"payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:21.202512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.186986 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb"] Nov 03 19:06:21.202512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.191951 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74b98\" (UniqueName: \"kubernetes.io/projected/ccc02de7-a8a0-4c12-b809-11034c3ca45e-kube-api-access-74b98\") pod \"payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:21.215336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.215303 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.243926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.243898 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb"] Nov 03 19:06:21.249875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.249849 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzl75\" (UniqueName: \"kubernetes.io/projected/44530f1b-e75a-4856-8b82-c7cc85440525-kube-api-access-vzl75\") pod \"payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.249980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.249916 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44530f1b-e75a-4856-8b82-c7cc85440525-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.351651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.351609 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzl75\" (UniqueName: \"kubernetes.io/projected/44530f1b-e75a-4856-8b82-c7cc85440525-kube-api-access-vzl75\") pod \"payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.352139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.351811 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44530f1b-e75a-4856-8b82-c7cc85440525-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.352979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.352945 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44530f1b-e75a-4856-8b82-c7cc85440525-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.388123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.388096 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:21.414206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.414173 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzl75\" (UniqueName: \"kubernetes.io/projected/44530f1b-e75a-4856-8b82-c7cc85440525-kube-api-access-vzl75\") pod \"payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.507614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.507575 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" event={"ID":"e5837ad6-6036-4611-aa0d-149b7f52c356","Type":"ContainerStarted","Data":"37afdacb68fd9d430ff34222a09a8709dae5f06a51f9aacf809fa67ff130f073"} Nov 03 19:06:21.539298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.539268 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:21.796472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.795585 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:21.824753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.824512 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9"] Nov 03 19:06:21.874732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.873677 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f475bdd3-4f45-4fdc-a884-0580f9450c75-kube-api-access-plkqb\" (UniqueName: \"kubernetes.io/projected/f475bdd3-4f45-4fdc-a884-0580f9450c75-kube-api-access-plkqb\") pod \"f475bdd3-4f45-4fdc-a884-0580f9450c75\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " Nov 03 19:06:21.874732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.873796 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-openshift-ca\") pod \"f475bdd3-4f45-4fdc-a884-0580f9450c75\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " Nov 03 19:06:21.874732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.873853 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-xks-ca\") pod \"f475bdd3-4f45-4fdc-a884-0580f9450c75\" (UID: \"f475bdd3-4f45-4fdc-a884-0580f9450c75\") " Nov 03 19:06:21.877500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.876117 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-openshift-ca" pod "f475bdd3-4f45-4fdc-a884-0580f9450c75" (UID: "f475bdd3-4f45-4fdc-a884-0580f9450c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:06:21.891721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.888949 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-xks-ca" pod "f475bdd3-4f45-4fdc-a884-0580f9450c75" (UID: "f475bdd3-4f45-4fdc-a884-0580f9450c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:06:21.891721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.890906 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f475bdd3-4f45-4fdc-a884-0580f9450c75-kube-api-access-plkqb" pod "f475bdd3-4f45-4fdc-a884-0580f9450c75" (UID: "f475bdd3-4f45-4fdc-a884-0580f9450c75"). InnerVolumeSpecName "kube-api-access-plkqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:06:21.977173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.977141 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plkqb\" (UniqueName: \"kubernetes.io/projected/f475bdd3-4f45-4fdc-a884-0580f9450c75-kube-api-access-plkqb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:21.977284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.977176 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:21.977284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:21.977192 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f475bdd3-4f45-4fdc-a884-0580f9450c75-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:22.012494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.011114 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb"] Nov 03 19:06:22.015967 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:06:22.015941 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44530f1b_e75a_4856_8b82_c7cc85440525.slice/crio-12a0c98508f98e52355c1614ef53bedd48ef93fa466f3291a4650076979ba069 WatchSource:0}: Error finding container 12a0c98508f98e52355c1614ef53bedd48ef93fa466f3291a4650076979ba069: Status 404 returned error can't find the container with id 12a0c98508f98e52355c1614ef53bedd48ef93fa466f3291a4650076979ba069 Nov 03 19:06:22.139581 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.139557 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:06:22.140732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.140691 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:06:22.140822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.140760 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 19:06:22.520333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.520304 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" Nov 03 19:06:22.520784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.520342 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b" event={"ID":"f475bdd3-4f45-4fdc-a884-0580f9450c75","Type":"ContainerDied","Data":"cbeea112e5dbdb5fcf398b7b490d778e4a51b82beb23343fd862d760a872aafe"} Nov 03 19:06:22.520784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.520402 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbeea112e5dbdb5fcf398b7b490d778e4a51b82beb23343fd862d760a872aafe" Nov 03 19:06:22.522528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.522499 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" event={"ID":"ccc02de7-a8a0-4c12-b809-11034c3ca45e","Type":"ContainerStarted","Data":"5969b6114e21fb44073e6f075b4fb3a26438c01ed3b52abf881c10ad7d518fe2"} Nov 03 19:06:22.522661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.522564 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" event={"ID":"ccc02de7-a8a0-4c12-b809-11034c3ca45e","Type":"ContainerStarted","Data":"3177183bdf165cfcb7bc14b60f731ab56bc3481ee7e6e2ec781b3e1fd9eccca9"} Nov 03 19:06:22.522876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.522854 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:22.525682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.525655 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" event={"ID":"44530f1b-e75a-4856-8b82-c7cc85440525","Type":"ContainerStarted","Data":"e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6"} Nov 03 19:06:22.525796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.525741 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" event={"ID":"44530f1b-e75a-4856-8b82-c7cc85440525","Type":"ContainerStarted","Data":"12a0c98508f98e52355c1614ef53bedd48ef93fa466f3291a4650076979ba069"} Nov 03 19:06:22.525796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.525768 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:22.650350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.650298 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" podStartSLOduration=2.650283477 podStartE2EDuration="2.650283477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:22.645310767 +0000 UTC m=+1712.128306683" watchObservedRunningTime="2026-11-03 19:06:22.650283477 +0000 UTC m=+1712.133279390" Nov 03 19:06:22.745096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.745061 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck"] Nov 03 19:06:22.745270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.745249 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" containerID="cri-o://6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff" gracePeriod=30 Nov 03 19:06:22.837623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:22.837527 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" podStartSLOduration=2.837499352 podStartE2EDuration="2.837499352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:22.836623478 +0000 UTC m=+1712.319619389" watchObservedRunningTime="2026-11-03 19:06:22.837499352 +0000 UTC m=+1712.320495265" Nov 03 19:06:23.139621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:23.139523 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:06:23.139621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:23.139594 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 19:06:24.096502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:24.094959 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4"] Nov 03 19:06:24.104151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:24.104115 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c31f13c8-swpw4"] Nov 03 19:06:24.140074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:24.140043 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:06:24.140235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:24.140093 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 19:06:24.790492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:24.790441 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e8dbf6-c74b-4371-bc3d-0e0a6288522b" path="/var/lib/kubelet/pods/51e8dbf6-c74b-4371-bc3d-0e0a6288522b/volumes" Nov 03 19:06:25.139446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:25.139364 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:06:25.139446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:25.139422 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 19:06:26.085190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:26.085157 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb"] Nov 03 19:06:26.110419 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:26.110380 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cde3da4-4ngfb"] Nov 03 19:06:26.140266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:26.140231 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:06:26.140618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:26.140287 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 19:06:26.783920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:26.783884 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7dd0e93-0807-42dd-a907-3c0796c68fdb" path="/var/lib/kubelet/pods/a7dd0e93-0807-42dd-a907-3c0796c68fdb/volumes" Nov 03 19:06:27.140315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:27.140239 4948 patch_prober.go:29] interesting pod/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:06:27.140673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:27.140309 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 19:06:27.297378 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:27.297342 4948 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:06:27.297542 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:27.297403 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:06:28.036007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.035939 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 19:06:28.144009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.143982 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:06:28.144360 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.144253 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:06:28.145093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.145072 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:06:28.190619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.190577 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-envoy\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.190619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.190630 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-credential-socket\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.190872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.190671 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-socket\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.190872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.190755 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-kube-api-access-9vrj7\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-kube-api-access-9vrj7\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.190990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.190915 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-podinfo\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.190990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.190945 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-token\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-token\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.191107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.190974 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-credential-socket" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:06:28.191107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.191030 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-socket" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:06:28.191107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.191039 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-certs\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.191259 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.191226 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-data\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.191259 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.191233 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-certs" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:06:28.191362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.191266 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istiod-ca-cert\") pod \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\" (UID: \"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19\") " Nov 03 19:06:28.191621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.191544 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-data" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:06:28.191830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.191806 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istiod-ca-cert" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:06:28.192229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.192207 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.192309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.192232 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.192309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.192248 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.192309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.192262 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.192309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.192276 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.193361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.193338 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-envoy" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:06:28.193443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.193368 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-podinfo" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:06:28.193562 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.193543 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-kube-api-access-9vrj7" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "kube-api-access-9vrj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:06:28.193792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.193768 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-token" pod "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" (UID: "92a0efd5-65a2-48d6-ad36-a4e24fd7ba19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:06:28.295545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.294389 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.295545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.294427 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrj7\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-kube-api-access-9vrj7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.295545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.294442 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.295545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.294465 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:28.614297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.614202 4948 generic.go:356] "Generic (PLEG): container finished" podID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerID="6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff" exitCode=0 Nov 03 19:06:28.614297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.614264 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" Nov 03 19:06:28.614498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.614256 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" event={"ID":"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19","Type":"ContainerDied","Data":"6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff"} Nov 03 19:06:28.614498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.614406 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck" event={"ID":"92a0efd5-65a2-48d6-ad36-a4e24fd7ba19","Type":"ContainerDied","Data":"0a6f9d8757bff660d5510a9e9585cc7b0176ef53175b3937f120716b6146a570"} Nov 03 19:06:28.614498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.614432 4948 scope.go:122] "RemoveContainer" containerID="6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff" Nov 03 19:06:28.628258 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.628236 4948 scope.go:122] "RemoveContainer" containerID="6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff" Nov 03 19:06:28.628510 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:06:28.628494 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff\": container with ID starting with 6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff not found: ID does not exist" containerID="6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff" Nov 03 19:06:28.628547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.628518 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff"} err="failed to get container status \"6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff\": rpc error: code = NotFound desc = could not find container \"6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff\": container with ID starting with 6fd457b0595f84e330cd8c9cdf4addf9242d895949dcd7ad221da907d1e9c3ff not found: ID does not exist" Nov 03 19:06:28.787897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.787859 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck"] Nov 03 19:06:28.838762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:28.838728 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-rf5ck"] Nov 03 19:06:29.058624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:29.058580 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68"] Nov 03 19:06:29.064428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:29.064400 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a78933-9ng68"] Nov 03 19:06:30.062013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:30.061978 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd"] Nov 03 19:06:30.113104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:30.113074 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d591c0-sh6rd"] Nov 03 19:06:30.757233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:30.757193 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b67b136-e248-4823-978a-5d355a8ef553" path="/var/lib/kubelet/pods/1b67b136-e248-4823-978a-5d355a8ef553/volumes" Nov 03 19:06:30.757882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:30.757856 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514e5e64-027b-4665-8508-560326c5a39b" path="/var/lib/kubelet/pods/514e5e64-027b-4665-8508-560326c5a39b/volumes" Nov 03 19:06:30.758443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:30.758420 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" path="/var/lib/kubelet/pods/92a0efd5-65a2-48d6-ad36-a4e24fd7ba19/volumes" Nov 03 19:06:32.797259 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:06:32.797214 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:06:33.539989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:33.539956 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:06:33.540422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:33.540399 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:06:36.111141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.111100 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr"] Nov 03 19:06:36.157015 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.156984 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm"] Nov 03 19:06:36.162766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.162728 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c6dc9bc-h86cm"] Nov 03 19:06:36.168053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.168023 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be375354-n9chr"] Nov 03 19:06:36.757300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.757260 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb29dd1-c13a-4805-9b60-521f3bf2a842" path="/var/lib/kubelet/pods/3eb29dd1-c13a-4805-9b60-521f3bf2a842/volumes" Nov 03 19:06:36.757926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.757900 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ff101f-eb04-48c3-9309-61bbd285a8fa" path="/var/lib/kubelet/pods/49ff101f-eb04-48c3-9309-61bbd285a8fa/volumes" Nov 03 19:06:36.916042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.916008 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d"] Nov 03 19:06:36.916816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.916801 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f475bdd3-4f45-4fdc-a884-0580f9450c75" containerName="revoke-keys" Nov 03 19:06:36.916816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.916816 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f475bdd3-4f45-4fdc-a884-0580f9450c75" containerName="revoke-keys" Nov 03 19:06:36.916901 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.916836 4948 cpu_manager.go:414] "containerMap: removing container" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" Nov 03 19:06:36.916901 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.916841 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" Nov 03 19:06:36.917000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.916989 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f475bdd3-4f45-4fdc-a884-0580f9450c75" containerName="revoke-keys" Nov 03 19:06:36.917036 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.917003 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a0efd5-65a2-48d6-ad36-a4e24fd7ba19" containerName="istio-proxy" Nov 03 19:06:36.921070 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.921046 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:36.956314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.956277 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:36.956476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.956425 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6r4d\" (UniqueName: \"kubernetes.io/projected/a06cf90c-7126-4d90-8226-76a262f2a316-kube-api-access-w6r4d\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:36.956680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.956651 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:36.979055 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:36.978567 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d"] Nov 03 19:06:37.019746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.019618 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:06:37.019746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.019684 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:06:37.019746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.019684 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:06:37.020018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.019744 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:06:37.058681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.058650 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:37.058896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.058879 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:37.059028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.059005 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6r4d\" (UniqueName: \"kubernetes.io/projected/a06cf90c-7126-4d90-8226-76a262f2a316-kube-api-access-w6r4d\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:37.059661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.059635 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:37.061225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.061201 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:37.121491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.121454 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6r4d\" (UniqueName: \"kubernetes.io/projected/a06cf90c-7126-4d90-8226-76a262f2a316-kube-api-access-w6r4d\") pod \"maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:37.239413 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.239379 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:37.641720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.632791 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d"] Nov 03 19:06:37.641720 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:06:37.633791 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06cf90c_7126_4d90_8226_76a262f2a316.slice/crio-6a16df92772b26f2ddd2e539488a82391eb25a7db29412ab3e154a73be5e1ec2 WatchSource:0}: Error finding container 6a16df92772b26f2ddd2e539488a82391eb25a7db29412ab3e154a73be5e1ec2: Status 404 returned error can't find the container with id 6a16df92772b26f2ddd2e539488a82391eb25a7db29412ab3e154a73be5e1ec2 Nov 03 19:06:37.711992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:37.711957 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" event={"ID":"a06cf90c-7126-4d90-8226-76a262f2a316","Type":"ContainerStarted","Data":"6a16df92772b26f2ddd2e539488a82391eb25a7db29412ab3e154a73be5e1ec2"} Nov 03 19:06:38.059470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:38.059436 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv"] Nov 03 19:06:38.104133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:38.104094 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aaa9e67-knpbv"] Nov 03 19:06:38.724076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:38.724040 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" event={"ID":"a06cf90c-7126-4d90-8226-76a262f2a316","Type":"ContainerStarted","Data":"4c6a34d7cf35f3114919d59c06e53462a2039e7fd07f753be8999ade0a02b242"} Nov 03 19:06:38.758286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:38.758253 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ff9ac2-580c-4828-9d05-a2b48c2b57d9" path="/var/lib/kubelet/pods/57ff9ac2-580c-4828-9d05-a2b48c2b57d9/volumes" Nov 03 19:06:38.846418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:38.846361 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" podStartSLOduration=2.846347475 podStartE2EDuration="2.846347475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:38.84199486 +0000 UTC m=+1728.324990775" watchObservedRunningTime="2026-11-03 19:06:38.846347475 +0000 UTC m=+1728.329343401" Nov 03 19:06:39.064275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:39.064186 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj"] Nov 03 19:06:39.069890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:39.069859 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8e329f-x4kbj"] Nov 03 19:06:39.737829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:39.737795 4948 generic.go:356] "Generic (PLEG): container finished" podID="a06cf90c-7126-4d90-8226-76a262f2a316" containerID="4c6a34d7cf35f3114919d59c06e53462a2039e7fd07f753be8999ade0a02b242" exitCode=0 Nov 03 19:06:39.738257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:39.737851 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" event={"ID":"a06cf90c-7126-4d90-8226-76a262f2a316","Type":"ContainerDied","Data":"4c6a34d7cf35f3114919d59c06e53462a2039e7fd07f753be8999ade0a02b242"} Nov 03 19:06:40.843724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:40.841230 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874390a6-68da-4a9c-9680-64cc1ebc2157" path="/var/lib/kubelet/pods/874390a6-68da-4a9c-9680-64cc1ebc2157/volumes" Nov 03 19:06:41.090092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.090059 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:41.172945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.171313 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-openshift-ca\") pod \"a06cf90c-7126-4d90-8226-76a262f2a316\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " Nov 03 19:06:41.172945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.171387 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-xks-ca\") pod \"a06cf90c-7126-4d90-8226-76a262f2a316\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " Nov 03 19:06:41.172945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.171533 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06cf90c-7126-4d90-8226-76a262f2a316-kube-api-access-w6r4d\" (UniqueName: \"kubernetes.io/projected/a06cf90c-7126-4d90-8226-76a262f2a316-kube-api-access-w6r4d\") pod \"a06cf90c-7126-4d90-8226-76a262f2a316\" (UID: \"a06cf90c-7126-4d90-8226-76a262f2a316\") " Nov 03 19:06:41.184862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.184801 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06cf90c-7126-4d90-8226-76a262f2a316-kube-api-access-w6r4d" pod "a06cf90c-7126-4d90-8226-76a262f2a316" (UID: "a06cf90c-7126-4d90-8226-76a262f2a316"). InnerVolumeSpecName "kube-api-access-w6r4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:06:41.185250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.185223 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-openshift-ca" pod "a06cf90c-7126-4d90-8226-76a262f2a316" (UID: "a06cf90c-7126-4d90-8226-76a262f2a316"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:06:41.187604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.187498 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-xks-ca" pod "a06cf90c-7126-4d90-8226-76a262f2a316" (UID: "a06cf90c-7126-4d90-8226-76a262f2a316"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:06:41.276729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.276238 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6r4d\" (UniqueName: \"kubernetes.io/projected/a06cf90c-7126-4d90-8226-76a262f2a316-kube-api-access-w6r4d\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:41.276729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.276281 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:41.276729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.276299 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06cf90c-7126-4d90-8226-76a262f2a316-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:41.893102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.893066 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" event={"ID":"a06cf90c-7126-4d90-8226-76a262f2a316","Type":"ContainerDied","Data":"6a16df92772b26f2ddd2e539488a82391eb25a7db29412ab3e154a73be5e1ec2"} Nov 03 19:06:41.893102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.893100 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d" Nov 03 19:06:41.893599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:41.893115 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a16df92772b26f2ddd2e539488a82391eb25a7db29412ab3e154a73be5e1ec2" Nov 03 19:06:42.073820 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:42.073759 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c"] Nov 03 19:06:42.094069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:42.094038 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1688ae35-z227c"] Nov 03 19:06:42.804451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:42.804406 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e056629f-76b0-49a7-b632-48a5729bf91f" path="/var/lib/kubelet/pods/e056629f-76b0-49a7-b632-48a5729bf91f/volumes" Nov 03 19:06:44.406767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.403215 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s"] Nov 03 19:06:44.406767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.404253 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a06cf90c-7126-4d90-8226-76a262f2a316" containerName="revoke-keys" Nov 03 19:06:44.406767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.404271 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06cf90c-7126-4d90-8226-76a262f2a316" containerName="revoke-keys" Nov 03 19:06:44.406767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.404401 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06cf90c-7126-4d90-8226-76a262f2a316" containerName="revoke-keys" Nov 03 19:06:44.411183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.411154 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.507779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.471777 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-58ac09-openshift-default-dockercfg-cgsjb\"" Nov 03 19:06:44.511729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.511645 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-kube-api-access-7mrqw\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.512445 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.512395 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-certs\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.512853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.512830 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-credential-socket\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.513133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.513102 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-data\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.513286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.513269 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-podinfo\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.513387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.513314 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99a5d5a-0550-45cf-8806-1daeb44e1730-istiod-ca-cert\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.513387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.513345 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-socket\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.513648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.513621 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-token\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.513780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.513752 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-envoy\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.531377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.531341 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s"] Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622255 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-token\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622346 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-envoy\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622396 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-kube-api-access-7mrqw\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622507 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-certs\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622649 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-credential-socket\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622725 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-data\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622769 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-podinfo\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622811 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99a5d5a-0550-45cf-8806-1daeb44e1730-istiod-ca-cert\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.622841 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-socket\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.623414 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-socket\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.648724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.630169 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-credential-socket\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.682777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.670677 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99a5d5a-0550-45cf-8806-1daeb44e1730-istiod-ca-cert\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.682777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.671636 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-certs\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.682777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.671933 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-data\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.843919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.835861 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-token\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.843919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.838846 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-podinfo\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.843919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.839097 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-envoy\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:44.926774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:44.919422 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-kube-api-access-7mrqw\") pod \"e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:45.052732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:45.047965 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:45.709967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:45.709932 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s"] Nov 03 19:06:45.743221 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:06:45.743189 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99a5d5a_0550_45cf_8806_1daeb44e1730.slice/crio-ce390483e2fef2864b6bd11b78245d77fa9289784ca8174b080e9e379f920d8f WatchSource:0}: Error finding container ce390483e2fef2864b6bd11b78245d77fa9289784ca8174b080e9e379f920d8f: Status 404 returned error can't find the container with id ce390483e2fef2864b6bd11b78245d77fa9289784ca8174b080e9e379f920d8f Nov 03 19:06:45.746045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:45.746010 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:45.746150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:45.746096 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:45.746150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:45.746142 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:46.011846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:46.011760 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" event={"ID":"b99a5d5a-0550-45cf-8806-1daeb44e1730","Type":"ContainerStarted","Data":"8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641"} Nov 03 19:06:46.011993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:46.011858 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" event={"ID":"b99a5d5a-0550-45cf-8806-1daeb44e1730","Type":"ContainerStarted","Data":"ce390483e2fef2864b6bd11b78245d77fa9289784ca8174b080e9e379f920d8f"} Nov 03 19:06:46.048938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:46.048901 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:46.050512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:46.050472 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 19:06:46.050600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:46.050544 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 19:06:46.142531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:46.142477 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podStartSLOduration=2.142461739 podStartE2EDuration="2.142461739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:46.138981038 +0000 UTC m=+1735.621976954" watchObservedRunningTime="2026-11-03 19:06:46.142461739 +0000 UTC m=+1735.625457631" Nov 03 19:06:47.093991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:47.088245 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:47.093991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:47.093759 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w"] Nov 03 19:06:47.123723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:47.122603 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7fd6b9-m9k2w"] Nov 03 19:06:48.036800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:48.036756 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:48.037849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:48.037824 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:06:48.541549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:48.541513 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4"] Nov 03 19:06:48.774569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:48.774531 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a8e82a-4018-4b2d-8555-3583e266313f" path="/var/lib/kubelet/pods/18a8e82a-4018-4b2d-8555-3583e266313f/volumes" Nov 03 19:06:50.173981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:50.173927 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:06:50.174669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:50.174636 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:52.142724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:52.132759 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v"] Nov 03 19:06:52.189490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:52.188087 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f09ce36-f526v"] Nov 03 19:06:52.830670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:52.830628 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefe3c15-f5cc-4845-bb2d-a5b41220b954" path="/var/lib/kubelet/pods/eefe3c15-f5cc-4845-bb2d-a5b41220b954/volumes" Nov 03 19:06:53.646724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:53.642769 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" podUID="c3c8e353-f35c-4d81-8a3a-2103adcacb91" containerName="payload-processing" containerID="cri-o://750dc569c6114292f73d1a2e2d2a20f0cef814676d2857aac565bebb5aa35f8b" gracePeriod=25 Nov 03 19:06:54.055638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.047783 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:54.214723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.211007 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:54.303725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.262072 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:54.344774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.304020 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 19:06:54.512730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.510831 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks"] Nov 03 19:06:54.632774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.630399 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:06:54.710175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.708819 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:54.895861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.895763 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84"] Nov 03 19:06:54.978798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:54.978755 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d8a519-mgfks"] Nov 03 19:06:55.049626 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.049426 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fee69a-z6p84"] Nov 03 19:06:55.748725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.737478 4948 generic.go:356] "Generic (PLEG): container finished" podID="c3c8e353-f35c-4d81-8a3a-2103adcacb91" containerID="750dc569c6114292f73d1a2e2d2a20f0cef814676d2857aac565bebb5aa35f8b" exitCode=0 Nov 03 19:06:55.748725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.737855 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" event={"ID":"c3c8e353-f35c-4d81-8a3a-2103adcacb91","Type":"ContainerDied","Data":"750dc569c6114292f73d1a2e2d2a20f0cef814676d2857aac565bebb5aa35f8b"} Nov 03 19:06:55.905333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.905302 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:06:55.947321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.947289 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c8e353-f35c-4d81-8a3a-2103adcacb91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c8e353-f35c-4d81-8a3a-2103adcacb91-plugins-config-volume\") pod \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " Nov 03 19:06:55.947527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.947507 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c8e353-f35c-4d81-8a3a-2103adcacb91-kube-api-access-c9pq9\" (UniqueName: \"kubernetes.io/projected/c3c8e353-f35c-4d81-8a3a-2103adcacb91-kube-api-access-c9pq9\") pod \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\" (UID: \"c3c8e353-f35c-4d81-8a3a-2103adcacb91\") " Nov 03 19:06:55.947811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.947785 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c8e353-f35c-4d81-8a3a-2103adcacb91-plugins-config-volume" pod "c3c8e353-f35c-4d81-8a3a-2103adcacb91" (UID: "c3c8e353-f35c-4d81-8a3a-2103adcacb91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:06:55.948193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.948171 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c8e353-f35c-4d81-8a3a-2103adcacb91-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:55.949878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:55.949855 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c8e353-f35c-4d81-8a3a-2103adcacb91-kube-api-access-c9pq9" pod "c3c8e353-f35c-4d81-8a3a-2103adcacb91" (UID: "c3c8e353-f35c-4d81-8a3a-2103adcacb91"). InnerVolumeSpecName "kube-api-access-c9pq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:06:56.050656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.050567 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pq9\" (UniqueName: \"kubernetes.io/projected/c3c8e353-f35c-4d81-8a3a-2103adcacb91-kube-api-access-c9pq9\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:56.335401 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:06:56.335371 4948 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e0c00c_5a75_46d0_8042_d43c2c776cd8.slice/crio-cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:06:56.588895 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.588788 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6"] Nov 03 19:06:56.589766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.589741 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c3c8e353-f35c-4d81-8a3a-2103adcacb91" containerName="payload-processing" Nov 03 19:06:56.589766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.589764 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c8e353-f35c-4d81-8a3a-2103adcacb91" containerName="payload-processing" Nov 03 19:06:56.589966 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.589950 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c8e353-f35c-4d81-8a3a-2103adcacb91" containerName="payload-processing" Nov 03 19:06:56.592861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.592839 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.646430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.646400 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:06:56.646605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.646476 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:06:56.652993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.651687 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6"] Nov 03 19:06:56.659190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.659151 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqgk\" (UniqueName: \"kubernetes.io/projected/e44b2cfb-1862-476f-9d40-259e440ba422-kube-api-access-7vqgk\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.659996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.659536 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.659996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.659639 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.695427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.695399 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:06:56.695600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.695489 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:06:56.754317 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.754287 4948 generic.go:356] "Generic (PLEG): container finished" podID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerID="cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37" exitCode=0 Nov 03 19:06:56.756428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.756403 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" Nov 03 19:06:56.757181 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.757154 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a07937-1758-4c74-bd78-9032f9606925" path="/var/lib/kubelet/pods/13a07937-1758-4c74-bd78-9032f9606925/volumes" Nov 03 19:06:56.757996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.757947 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a08a1a7-f03d-4932-a580-073ce53525c1" path="/var/lib/kubelet/pods/3a08a1a7-f03d-4932-a580-073ce53525c1/volumes" Nov 03 19:06:56.758797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.758775 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" event={"ID":"23e0c00c-5a75-46d0-8042-d43c2c776cd8","Type":"ContainerDied","Data":"cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37"} Nov 03 19:06:56.758863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.758828 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4" event={"ID":"c3c8e353-f35c-4d81-8a3a-2103adcacb91","Type":"ContainerDied","Data":"a77462f22e853935bfc645aa1028eb8f1c42d190f44e274b0eaecada1465bbc1"} Nov 03 19:06:56.758919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.758865 4948 scope.go:122] "RemoveContainer" containerID="750dc569c6114292f73d1a2e2d2a20f0cef814676d2857aac565bebb5aa35f8b" Nov 03 19:06:56.759361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.759343 4948 scope.go:122] "RemoveContainer" containerID="cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37" Nov 03 19:06:56.760806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.760779 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.760926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.760874 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.760926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.760916 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqgk\" (UniqueName: \"kubernetes.io/projected/e44b2cfb-1862-476f-9d40-259e440ba422-kube-api-access-7vqgk\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.761746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.761716 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.764155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.764117 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.764924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.764893 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:56.765031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.764978 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:56.765097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.765030 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:06:56.823674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.823644 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqgk\" (UniqueName: \"kubernetes.io/projected/e44b2cfb-1862-476f-9d40-259e440ba422-kube-api-access-7vqgk\") pod \"maas-api-revoke-keys-e2e-ait-6765e083-x88q6\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.907763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.907733 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:06:56.930511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.930464 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4"] Nov 03 19:06:56.936388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:56.936297 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-82cd41ab-f78868cf5-thnf4"] Nov 03 19:06:57.324659 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:06:57.324624 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44b2cfb_1862_476f_9d40_259e440ba422.slice/crio-64ff70137f72462f38f1d28f9abf7f8bf785cff6830881846ce454d207d867b5 WatchSource:0}: Error finding container 64ff70137f72462f38f1d28f9abf7f8bf785cff6830881846ce454d207d867b5: Status 404 returned error can't find the container with id 64ff70137f72462f38f1d28f9abf7f8bf785cff6830881846ce454d207d867b5 Nov 03 19:06:57.328080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:57.328056 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6"] Nov 03 19:06:57.797798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:57.797764 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" event={"ID":"23e0c00c-5a75-46d0-8042-d43c2c776cd8","Type":"ContainerStarted","Data":"7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726"} Nov 03 19:06:57.801939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:57.801899 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" event={"ID":"e44b2cfb-1862-476f-9d40-259e440ba422","Type":"ContainerStarted","Data":"64ff70137f72462f38f1d28f9abf7f8bf785cff6830881846ce454d207d867b5"} Nov 03 19:06:58.087420 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.087328 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:06:58.087883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.087858 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:06:58.087974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.087910 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 19:06:58.342670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.342592 4948 scope.go:122] "RemoveContainer" containerID="e5892109a42ca92d4bd54c8c8bb213b62c270577a4fbc271e9ffff82fad4992a" Nov 03 19:06:58.386982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.386954 4948 scope.go:122] "RemoveContainer" containerID="d6423b1b974dff960315f029b7a2e1a65c98756bd2ba9afabaa7c4e46676ba56" Nov 03 19:06:58.401303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.401273 4948 scope.go:122] "RemoveContainer" containerID="2dfdbe69cf66ebd677f56e03fae5881a09259b58ac0ff06411ee2d4d9a27b325" Nov 03 19:06:58.412950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.412932 4948 scope.go:122] "RemoveContainer" containerID="87aff9994e7f8dc7daa0ed757061666aa5fc7602e7fe5f3f18de2bd02165da3c" Nov 03 19:06:58.426707 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.426678 4948 scope.go:122] "RemoveContainer" containerID="7ab1982cc1e6aa7dfea5d92e81ce5f68a9820a3b27cb3e1f823582e305a55cf1" Nov 03 19:06:58.784935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.784900 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c8e353-f35c-4d81-8a3a-2103adcacb91" path="/var/lib/kubelet/pods/c3c8e353-f35c-4d81-8a3a-2103adcacb91/volumes" Nov 03 19:06:58.812947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.812913 4948 generic.go:356] "Generic (PLEG): container finished" podID="e44b2cfb-1862-476f-9d40-259e440ba422" containerID="1fd3a75f1b367b821de9b7c6cc2277b34fc6097753077cd51aeb6ec5587ba339" exitCode=0 Nov 03 19:06:58.813322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:58.812998 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" event={"ID":"e44b2cfb-1862-476f-9d40-259e440ba422","Type":"ContainerDied","Data":"1fd3a75f1b367b821de9b7c6cc2277b34fc6097753077cd51aeb6ec5587ba339"} Nov 03 19:06:59.086806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:59.086767 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:06:59.086981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:06:59.086821 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 19:07:00.046410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.046384 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:07:00.049653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.049627 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-xks-ca\") pod \"e44b2cfb-1862-476f-9d40-259e440ba422\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " Nov 03 19:07:00.049912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.049895 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-openshift-ca\") pod \"e44b2cfb-1862-476f-9d40-259e440ba422\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " Nov 03 19:07:00.049997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.049982 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44b2cfb-1862-476f-9d40-259e440ba422-kube-api-access-7vqgk\" (UniqueName: \"kubernetes.io/projected/e44b2cfb-1862-476f-9d40-259e440ba422-kube-api-access-7vqgk\") pod \"e44b2cfb-1862-476f-9d40-259e440ba422\" (UID: \"e44b2cfb-1862-476f-9d40-259e440ba422\") " Nov 03 19:07:00.050283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.050259 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-openshift-ca" pod "e44b2cfb-1862-476f-9d40-259e440ba422" (UID: "e44b2cfb-1862-476f-9d40-259e440ba422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:00.074723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.050751 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:00.079496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.078822 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-xks-ca" pod "e44b2cfb-1862-476f-9d40-259e440ba422" (UID: "e44b2cfb-1862-476f-9d40-259e440ba422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:07:00.080720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.080623 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44b2cfb-1862-476f-9d40-259e440ba422-kube-api-access-7vqgk" pod "e44b2cfb-1862-476f-9d40-259e440ba422" (UID: "e44b2cfb-1862-476f-9d40-259e440ba422"). InnerVolumeSpecName "kube-api-access-7vqgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:00.086753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.086177 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:07:00.086753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.086234 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 19:07:00.181724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.177974 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqgk\" (UniqueName: \"kubernetes.io/projected/e44b2cfb-1862-476f-9d40-259e440ba422-kube-api-access-7vqgk\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:00.181724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.178007 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44b2cfb-1862-476f-9d40-259e440ba422-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:00.842110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.841545 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" Nov 03 19:07:00.842110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.841574 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6" event={"ID":"e44b2cfb-1862-476f-9d40-259e440ba422","Type":"ContainerDied","Data":"64ff70137f72462f38f1d28f9abf7f8bf785cff6830881846ce454d207d867b5"} Nov 03 19:07:00.842110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:00.841635 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ff70137f72462f38f1d28f9abf7f8bf785cff6830881846ce454d207d867b5" Nov 03 19:07:01.086299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:01.086208 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:07:01.086299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:01.086280 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 19:07:02.085827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:02.085785 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:07:02.086020 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:02.085864 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 19:07:02.927661 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:02.927628 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:07:03.086218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:03.086178 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:07:03.086400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:03.086240 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 19:07:04.085086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:04.085052 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:07:04.091079 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:04.091046 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:07:04.958536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:04.958508 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:07:09.065320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:09.065282 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62"] Nov 03 19:07:09.116189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:09.116156 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa5f68d-5mf62"] Nov 03 19:07:10.061189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:10.061154 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548"] Nov 03 19:07:10.111266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:10.111230 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8d3e1d-t2548"] Nov 03 19:07:10.756170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:10.756133 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf9b5ac-92fe-4a21-b6b1-290321440d0c" path="/var/lib/kubelet/pods/eaf9b5ac-92fe-4a21-b6b1-290321440d0c/volumes" Nov 03 19:07:10.756712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:10.756673 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc51c215-341b-4565-9dbb-5c5493f51937" path="/var/lib/kubelet/pods/fc51c215-341b-4565-9dbb-5c5493f51937/volumes" Nov 03 19:07:12.057219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:12.057189 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz"] Nov 03 19:07:12.065525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:12.065495 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b0b2e2-6gfnz"] Nov 03 19:07:12.626227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:12.626194 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8"] Nov 03 19:07:12.757568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:12.757530 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa225ba-ef5e-470f-9cd2-c3911e843663" path="/var/lib/kubelet/pods/daa225ba-ef5e-470f-9cd2-c3911e843663/volumes" Nov 03 19:07:12.758083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:12.758061 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz"] Nov 03 19:07:13.253316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:13.253243 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb"] Nov 03 19:07:13.260763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:13.260711 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e43a950-rlctb"] Nov 03 19:07:14.813487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:14.813443 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41375853-6b74-44b4-8b99-0cd15393831f" path="/var/lib/kubelet/pods/41375853-6b74-44b4-8b99-0cd15393831f/volumes" Nov 03 19:07:17.635722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:17.628765 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" podUID="a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" containerName="payload-processing" containerID="cri-o://7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55" gracePeriod=25 Nov 03 19:07:17.786742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:17.784456 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" podUID="e3097574-7365-4487-9838-fe8fc220385b" containerName="payload-pre-processing" containerID="cri-o://95ec0a01cb7bdbb996c0e36c456ebeea33b7ba5157e7eb3d72d7cceac9e8ed19" gracePeriod=25 Nov 03 19:07:18.089643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.089611 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:07:18.161825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.161795 4948 generic.go:356] "Generic (PLEG): container finished" podID="e3097574-7365-4487-9838-fe8fc220385b" containerID="95ec0a01cb7bdbb996c0e36c456ebeea33b7ba5157e7eb3d72d7cceac9e8ed19" exitCode=0 Nov 03 19:07:18.161956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.161866 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" event={"ID":"e3097574-7365-4487-9838-fe8fc220385b","Type":"ContainerDied","Data":"95ec0a01cb7bdbb996c0e36c456ebeea33b7ba5157e7eb3d72d7cceac9e8ed19"} Nov 03 19:07:18.163614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.163590 4948 generic.go:356] "Generic (PLEG): container finished" podID="a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" containerID="7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55" exitCode=0 Nov 03 19:07:18.163823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.163804 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" event={"ID":"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8","Type":"ContainerDied","Data":"7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55"} Nov 03 19:07:18.163890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.163848 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" event={"ID":"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8","Type":"ContainerDied","Data":"775d85766c2ffdc61165f2e33b56c776d1b29d5613c2332cbaf8f211943335fa"} Nov 03 19:07:18.163890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.163873 4948 scope.go:122] "RemoveContainer" containerID="7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55" Nov 03 19:07:18.164023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.164012 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8" Nov 03 19:07:18.176532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.176512 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:07:18.177410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.177393 4948 scope.go:122] "RemoveContainer" containerID="7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55" Nov 03 19:07:18.177718 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:18.177686 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55\": container with ID starting with 7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55 not found: ID does not exist" containerID="7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55" Nov 03 19:07:18.177794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.177729 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55"} err="failed to get container status \"7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55\": rpc error: code = NotFound desc = could not find container \"7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55\": container with ID starting with 7a0917d32003cb72bbcec8a6846753cac503aa579d96b61d83f76d788d0d3d55 not found: ID does not exist" Nov 03 19:07:18.217301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.217275 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-kube-api-access-ks49h\" (UniqueName: \"kubernetes.io/projected/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-kube-api-access-ks49h\") pod \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " Nov 03 19:07:18.217507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.217485 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-plugins-config-volume\") pod \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\" (UID: \"a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8\") " Nov 03 19:07:18.217914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.217887 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-plugins-config-volume" pod "a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" (UID: "a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:18.218339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.218314 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:18.220020 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.219983 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-kube-api-access-ks49h" pod "a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" (UID: "a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8"). InnerVolumeSpecName "kube-api-access-ks49h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:18.319065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.318966 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3097574-7365-4487-9838-fe8fc220385b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3097574-7365-4487-9838-fe8fc220385b-plugins-config-volume\") pod \"e3097574-7365-4487-9838-fe8fc220385b\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " Nov 03 19:07:18.319253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.319142 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3097574-7365-4487-9838-fe8fc220385b-kube-api-access-dc897\" (UniqueName: \"kubernetes.io/projected/e3097574-7365-4487-9838-fe8fc220385b-kube-api-access-dc897\") pod \"e3097574-7365-4487-9838-fe8fc220385b\" (UID: \"e3097574-7365-4487-9838-fe8fc220385b\") " Nov 03 19:07:18.319393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.319363 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3097574-7365-4487-9838-fe8fc220385b-plugins-config-volume" pod "e3097574-7365-4487-9838-fe8fc220385b" (UID: "e3097574-7365-4487-9838-fe8fc220385b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:18.320041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.320012 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks49h\" (UniqueName: \"kubernetes.io/projected/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8-kube-api-access-ks49h\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:18.320041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.320042 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3097574-7365-4487-9838-fe8fc220385b-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:18.321576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.321549 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3097574-7365-4487-9838-fe8fc220385b-kube-api-access-dc897" pod "e3097574-7365-4487-9838-fe8fc220385b" (UID: "e3097574-7365-4487-9838-fe8fc220385b"). InnerVolumeSpecName "kube-api-access-dc897". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:18.422329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.422231 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc897\" (UniqueName: \"kubernetes.io/projected/e3097574-7365-4487-9838-fe8fc220385b-kube-api-access-dc897\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:18.594192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.594156 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8"] Nov 03 19:07:18.603634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.603584 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6765e083-55c596cf9f-nrbv8"] Nov 03 19:07:18.780604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:18.780566 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" path="/var/lib/kubelet/pods/a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8/volumes" Nov 03 19:07:19.230731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:19.228941 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" Nov 03 19:07:19.239067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:19.231174 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz" event={"ID":"e3097574-7365-4487-9838-fe8fc220385b","Type":"ContainerDied","Data":"ca3c164b73e8ba286fad938f0ff80bca727a8924d8fb374a642dc2bb2ab91de5"} Nov 03 19:07:19.239067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:19.231276 4948 scope.go:122] "RemoveContainer" containerID="95ec0a01cb7bdbb996c0e36c456ebeea33b7ba5157e7eb3d72d7cceac9e8ed19" Nov 03 19:07:19.418413 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:19.418351 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz"] Nov 03 19:07:19.427991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:19.427960 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6765e083-6df79b4b87-jx2qz"] Nov 03 19:07:20.780724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:20.780265 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3097574-7365-4487-9838-fe8fc220385b" path="/var/lib/kubelet/pods/e3097574-7365-4487-9838-fe8fc220385b/volumes" Nov 03 19:07:22.203922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:22.203872 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg"] Nov 03 19:07:22.214967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:22.213938 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb7e57a2-fzbgg"] Nov 03 19:07:22.826582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:22.826538 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2a8780-28fd-4595-a86b-0be48526c486" path="/var/lib/kubelet/pods/8c2a8780-28fd-4595-a86b-0be48526c486/volumes" Nov 03 19:07:24.088893 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:24.086248 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7"] Nov 03 19:07:24.104355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:24.104304 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c878b2f-4s5c7"] Nov 03 19:07:24.760614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:24.760567 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4df7583-55ad-4641-96dc-49b643910558" path="/var/lib/kubelet/pods/c4df7583-55ad-4641-96dc-49b643910558/volumes" Nov 03 19:07:27.060219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:27.060177 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj"] Nov 03 19:07:27.107968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:27.107933 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce6493-w7qrj"] Nov 03 19:07:27.198866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:27.198822 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:07:27.199059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:27.198894 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:07:28.756629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:28.756595 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96dac90e-7f3c-4db1-b930-6c578ff38b59" path="/var/lib/kubelet/pods/96dac90e-7f3c-4db1-b930-6c578ff38b59/volumes" Nov 03 19:07:29.494569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:29.494538 4948 generic.go:356] "Generic (PLEG): container finished" podID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerID="17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95" exitCode=0 Nov 03 19:07:29.494763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:29.494618 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" event={"ID":"ac010993-8cde-4557-a9fe-ec54ab8dc11b","Type":"ContainerDied","Data":"17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95"} Nov 03 19:07:29.495170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:29.495152 4948 scope.go:122] "RemoveContainer" containerID="17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95" Nov 03 19:07:29.497004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:29.496972 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:29.497129 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:29.497056 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:29.497129 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:29.497103 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:30.510763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:30.510686 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" event={"ID":"ac010993-8cde-4557-a9fe-ec54ab8dc11b","Type":"ContainerStarted","Data":"0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b"} Nov 03 19:07:30.772939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:30.772836 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 19:07:30.773233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:30.773199 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 19:07:30.786726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:30.781223 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:07:31.749366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:31.749328 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 19:07:31.749790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:31.749395 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 19:07:32.749027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:32.748990 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 19:07:32.749204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:32.749059 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 19:07:32.820328 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:32.820283 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:07:33.749003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:33.748966 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 19:07:33.749178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:33.749026 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 19:07:34.748800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:34.748758 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 19:07:34.749210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:34.748828 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 19:07:34.998690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:34.998651 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d"] Nov 03 19:07:35.041163 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:35.040609 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-82cd41ab-h2v2d"] Nov 03 19:07:35.047607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:35.047573 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn"] Nov 03 19:07:35.047989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:35.047932 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" containerID="cri-o://37afdacb68fd9d430ff34222a09a8709dae5f06a51f9aacf809fa67ff130f073" gracePeriod=30 Nov 03 19:07:35.749217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:35.749181 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 19:07:35.749604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:35.749230 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 19:07:36.877626 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:36.877586 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06cf90c-7126-4d90-8226-76a262f2a316" path="/var/lib/kubelet/pods/a06cf90c-7126-4d90-8226-76a262f2a316/volumes" Nov 03 19:07:37.023265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.023233 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:07:37.025529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.025504 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:07:37.031667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.031633 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:07:37.258357 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.258321 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2"] Nov 03 19:07:37.258622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.258571 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" containerID="cri-o://b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640" gracePeriod=30 Nov 03 19:07:37.270729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.266800 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6"] Nov 03 19:07:37.270729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.267023 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" containerID="cri-o://9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9" gracePeriod=30 Nov 03 19:07:37.282413 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.282389 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh"] Nov 03 19:07:37.282844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.282813 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" containerID="cri-o://bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2" gracePeriod=30 Nov 03 19:07:37.287053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.287028 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb"] Nov 03 19:07:37.287243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.287218 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" containerID="cri-o://107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3" gracePeriod=30 Nov 03 19:07:37.785346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.785312 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8"] Nov 03 19:07:37.785609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:37.785579 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" containerID="cri-o://e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91" gracePeriod=30 Nov 03 19:07:40.098471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.098428 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:07:40.098975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.098506 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:07:40.118004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.117960 4948 generic.go:356] "Generic (PLEG): container finished" podID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerID="37afdacb68fd9d430ff34222a09a8709dae5f06a51f9aacf809fa67ff130f073" exitCode=0 Nov 03 19:07:40.118205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.118036 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" event={"ID":"e5837ad6-6036-4611-aa0d-149b7f52c356","Type":"ContainerDied","Data":"37afdacb68fd9d430ff34222a09a8709dae5f06a51f9aacf809fa67ff130f073"} Nov 03 19:07:40.118205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.118118 4948 scope.go:122] "RemoveContainer" containerID="e90fffd5428ef2ccb60ad4799b452a6724554f3f4b4f69f2a951158d23cebcc0" Nov 03 19:07:40.407985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.407957 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:07:40.439659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.439625 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5837ad6-6036-4611-aa0d-149b7f52c356-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5837ad6-6036-4611-aa0d-149b7f52c356-istiod-ca-cert\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.439820 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.439688 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-kube-api-access-5mfwg\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-kube-api-access-5mfwg\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.439820 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.439767 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-token\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-token\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.439930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.439848 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-envoy\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.439930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.439896 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-credential-socket\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.440046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.439941 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-data\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.440106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.440048 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5837ad6-6036-4611-aa0d-149b7f52c356-istiod-ca-cert" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:40.440106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.440066 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-certs\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.440221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.440108 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-podinfo\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.440221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.440214 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-socket\") pod \"e5837ad6-6036-4611-aa0d-149b7f52c356\" (UID: \"e5837ad6-6036-4611-aa0d-149b7f52c356\") " Nov 03 19:07:40.440423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.440385 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-certs" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:40.440635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.440613 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-socket" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:40.440758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.440630 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-data" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:40.441093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.441072 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.441144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.441103 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.441144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.441122 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.441144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.441138 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5837ad6-6036-4611-aa0d-149b7f52c356-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.441411 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.441386 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-credential-socket" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:40.442570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.442541 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-envoy" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:40.443124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.443098 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-kube-api-access-5mfwg" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "kube-api-access-5mfwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:40.443335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.443309 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-podinfo" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:40.443570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.443548 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-token" pod "e5837ad6-6036-4611-aa0d-149b7f52c356" (UID: "e5837ad6-6036-4611-aa0d-149b7f52c356"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:40.542738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.542695 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mfwg\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-kube-api-access-5mfwg\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.542738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.542735 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.542738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.542745 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.542961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.542755 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5837ad6-6036-4611-aa0d-149b7f52c356-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:40.542961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:40.542765 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5837ad6-6036-4611-aa0d-149b7f52c356-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:41.062781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.062658 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75"] Nov 03 19:07:41.067550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.067520 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7397a1cf-fwh75"] Nov 03 19:07:41.131303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.131275 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" Nov 03 19:07:41.131303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.131298 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn" event={"ID":"e5837ad6-6036-4611-aa0d-149b7f52c356","Type":"ContainerDied","Data":"2646aa67025f1f61f7243a6b3b66d546795caf2f4dd7f7df1358913e20f31bf4"} Nov 03 19:07:41.131826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.131360 4948 scope.go:122] "RemoveContainer" containerID="37afdacb68fd9d430ff34222a09a8709dae5f06a51f9aacf809fa67ff130f073" Nov 03 19:07:41.309159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.309127 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn"] Nov 03 19:07:41.359560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.359483 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-82cd41ab-gw-openshift-default-7c64cf7b5-x82rn"] Nov 03 19:07:41.641427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.641345 4948 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:07:41.641597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:41.641419 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:07:42.635556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.633517 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 19:07:42.686310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.685372 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-data\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.685478 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-certs\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.685624 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-podinfo\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.685660 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-data" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.686310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.686093 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-certs" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.686310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.686248 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-credential-socket\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.686400 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-envoy\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.686440 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-token\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-token\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.686491 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-kube-api-access-m5mp7\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-kube-api-access-m5mp7\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.686531 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-socket\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.686684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.686567 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istiod-ca-cert\") pod \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\" (UID: \"d3bed544-d72a-48d8-94fc-9c4f9707d16f\") " Nov 03 19:07:42.687216 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.687189 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-credential-socket" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.687465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.687443 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.687561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.687494 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.687561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.687514 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.687646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.687561 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-socket" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.688010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.687988 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 19:07:42.689891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.689866 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-envoy" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.689986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.689881 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-token" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.692831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.692809 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-podinfo" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:42.692957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.692943 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istiod-ca-cert" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:42.693827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.693807 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:07:42.696092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.696067 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-kube-api-access-m5mp7" pod "d3bed544-d72a-48d8-94fc-9c4f9707d16f" (UID: "d3bed544-d72a-48d8-94fc-9c4f9707d16f"). InnerVolumeSpecName "kube-api-access-m5mp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.742762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.742739 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:07:42.756029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.756004 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48157e9e-c9b0-4372-8167-9df5b0c6b9c7" path="/var/lib/kubelet/pods/48157e9e-c9b0-4372-8167-9df5b0c6b9c7/volumes" Nov 03 19:07:42.756590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.756575 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" path="/var/lib/kubelet/pods/e5837ad6-6036-4611-aa0d-149b7f52c356/volumes" Nov 03 19:07:42.788752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.788681 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-data\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.788883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.788785 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-envoy\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.788883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.788827 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-credential-socket\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.788975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.788883 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-envoy\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.788975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.788918 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-socket\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.788975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.788969 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-socket\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.789132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.788992 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-data" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.789132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789015 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d283713d-1736-4544-92a8-618ffc320af5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d283713d-1736-4544-92a8-618ffc320af5-istiod-ca-cert\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.789132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789055 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-certs\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.789132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789089 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-token\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-token\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.789344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789145 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-credential-socket" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.789344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789155 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-data\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.789344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789289 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-data\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.789344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789310 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-socket" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.789528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789353 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-certs" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.789528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789383 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-data" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.789528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789418 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-istio-token\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-istio-token\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.789528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789469 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-socket\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789557 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-socket" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789562 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-certs\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789578 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-data" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789588 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d283713d-1736-4544-92a8-618ffc320af5-istiod-ca-cert" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789602 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-kube-api-access-cjqhq\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-kube-api-access-cjqhq\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789651 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-credential-socket\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789732 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182a69ca-cec8-4640-82cd-04b021bea8e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a69ca-cec8-4640-82cd-04b021bea8e5-istiod-ca-cert\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789771 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-podinfo\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789788 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-socket" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789807 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b20e356e-72df-49be-9213-ac45383f3950-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e356e-72df-49be-9213-ac45383f3950-istio-podinfo\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789867 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-istio-token\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-istio-token\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789869 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-certs" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789910 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-kube-api-access-pqk52\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-kube-api-access-pqk52\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789968 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20e356e-72df-49be-9213-ac45383f3950-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e356e-72df-49be-9213-ac45383f3950-istiod-ca-cert\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.789998 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-credential-socket" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.790034 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-kube-api-access-k27gr\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-kube-api-access-k27gr\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.790066 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-envoy\") pod \"182a69ca-cec8-4640-82cd-04b021bea8e5\" (UID: \"182a69ca-cec8-4640-82cd-04b021bea8e5\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.790098 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d283713d-1736-4544-92a8-618ffc320af5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d283713d-1736-4544-92a8-618ffc320af5-istio-podinfo\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.790152 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-credential-socket\") pod \"d283713d-1736-4544-92a8-618ffc320af5\" (UID: \"d283713d-1736-4544-92a8-618ffc320af5\") " Nov 03 19:07:42.790293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.790199 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-certs\") pod \"b20e356e-72df-49be-9213-ac45383f3950\" (UID: \"b20e356e-72df-49be-9213-ac45383f3950\") " Nov 03 19:07:42.791184 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.791035 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182a69ca-cec8-4640-82cd-04b021bea8e5-istiod-ca-cert" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:42.792198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792176 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792324 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792434 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d283713d-1736-4544-92a8-618ffc320af5-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792541 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792530 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792663 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792782 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792879 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792917 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5mp7\" (UniqueName: \"kubernetes.io/projected/d3bed544-d72a-48d8-94fc-9c4f9707d16f-kube-api-access-m5mp7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792937 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.792962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792952 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3bed544-d72a-48d8-94fc-9c4f9707d16f-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792966 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792981 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792997 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.793011 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.793025 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182a69ca-cec8-4640-82cd-04b021bea8e5-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.793038 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3bed544-d72a-48d8-94fc-9c4f9707d16f-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.793051 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.793175 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.793491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792383 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-envoy" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.793491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792755 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-podinfo" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:42.793491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792773 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b20e356e-72df-49be-9213-ac45383f3950-istio-podinfo" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:42.793491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.792954 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20e356e-72df-49be-9213-ac45383f3950-istiod-ca-cert" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:42.793491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.793336 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-credential-socket" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.793858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.793622 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-certs" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.794042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.794017 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-istio-token" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.794918 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.794891 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-kube-api-access-cjqhq" pod "b20e356e-72df-49be-9213-ac45383f3950" (UID: "b20e356e-72df-49be-9213-ac45383f3950"). InnerVolumeSpecName "kube-api-access-cjqhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.795024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.795003 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-kube-api-access-pqk52" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "kube-api-access-pqk52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.795191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.795167 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-token" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.795575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.795544 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-kube-api-access-k27gr" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "kube-api-access-k27gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.795788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.795765 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-envoy" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.795884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.795792 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-istio-token" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.796093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.796074 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-envoy" pod "182a69ca-cec8-4640-82cd-04b021bea8e5" (UID: "182a69ca-cec8-4640-82cd-04b021bea8e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.796410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.796386 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d283713d-1736-4544-92a8-618ffc320af5-istio-podinfo" pod "d283713d-1736-4544-92a8-618ffc320af5" (UID: "d283713d-1736-4544-92a8-618ffc320af5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:42.895340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895309 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895338 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjqhq\" (UniqueName: \"kubernetes.io/projected/b20e356e-72df-49be-9213-ac45383f3950-kube-api-access-cjqhq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895349 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895358 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e356e-72df-49be-9213-ac45383f3950-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895370 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895385 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqk52\" (UniqueName: \"kubernetes.io/projected/d283713d-1736-4544-92a8-618ffc320af5-kube-api-access-pqk52\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895397 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e356e-72df-49be-9213-ac45383f3950-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895408 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27gr\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-kube-api-access-k27gr\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895417 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895428 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d283713d-1736-4544-92a8-618ffc320af5-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895436 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895445 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895461 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d283713d-1736-4544-92a8-618ffc320af5-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895474 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e356e-72df-49be-9213-ac45383f3950-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.895618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:42.895483 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182a69ca-cec8-4640-82cd-04b021bea8e5-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.133896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.133863 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:07:43.154032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.154006 4948 generic.go:356] "Generic (PLEG): container finished" podID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerID="b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640" exitCode=0 Nov 03 19:07:43.154149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.154049 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" Nov 03 19:07:43.154149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.154084 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" event={"ID":"182a69ca-cec8-4640-82cd-04b021bea8e5","Type":"ContainerDied","Data":"b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640"} Nov 03 19:07:43.154149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.154142 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2" event={"ID":"182a69ca-cec8-4640-82cd-04b021bea8e5","Type":"ContainerDied","Data":"d74962b518c1f205abd08d96548380ff45735a408f4aef8a52f010fd9ffea56e"} Nov 03 19:07:43.154285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.154178 4948 scope.go:122] "RemoveContainer" containerID="b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640" Nov 03 19:07:43.155742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.155717 4948 generic.go:356] "Generic (PLEG): container finished" podID="d283713d-1736-4544-92a8-618ffc320af5" containerID="107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3" exitCode=0 Nov 03 19:07:43.155884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.155788 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" Nov 03 19:07:43.155884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.155854 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" event={"ID":"d283713d-1736-4544-92a8-618ffc320af5","Type":"ContainerDied","Data":"107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3"} Nov 03 19:07:43.155986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.155900 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb" event={"ID":"d283713d-1736-4544-92a8-618ffc320af5","Type":"ContainerDied","Data":"de259e1db36899ce147c1bbad487b527d37bc8ad668e130a59011f8364f8cccf"} Nov 03 19:07:43.157665 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.157640 4948 generic.go:356] "Generic (PLEG): container finished" podID="b20e356e-72df-49be-9213-ac45383f3950" containerID="9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9" exitCode=0 Nov 03 19:07:43.157806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.157682 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" Nov 03 19:07:43.157867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.157820 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" event={"ID":"b20e356e-72df-49be-9213-ac45383f3950","Type":"ContainerDied","Data":"9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9"} Nov 03 19:07:43.157867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.157862 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6" event={"ID":"b20e356e-72df-49be-9213-ac45383f3950","Type":"ContainerDied","Data":"8e898ecc5e9d864ad244c2b6fa650925d314a67bb19f832013542c1dba35ce47"} Nov 03 19:07:43.159674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.159651 4948 generic.go:356] "Generic (PLEG): container finished" podID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerID="bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2" exitCode=0 Nov 03 19:07:43.159824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.159742 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" Nov 03 19:07:43.159824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.159777 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" event={"ID":"d3bed544-d72a-48d8-94fc-9c4f9707d16f","Type":"ContainerDied","Data":"bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2"} Nov 03 19:07:43.159824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.159814 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh" event={"ID":"d3bed544-d72a-48d8-94fc-9c4f9707d16f","Type":"ContainerDied","Data":"7bc611c9a6d9e24b6bdad16960d33a9be143226e8d57cae7d42a65eed4f7e638"} Nov 03 19:07:43.163531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.163509 4948 generic.go:356] "Generic (PLEG): container finished" podID="a659e245-b62d-4586-ac00-205ba64f6842" containerID="e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91" exitCode=0 Nov 03 19:07:43.163751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.163723 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" event={"ID":"a659e245-b62d-4586-ac00-205ba64f6842","Type":"ContainerDied","Data":"e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91"} Nov 03 19:07:43.163834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.163770 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" event={"ID":"a659e245-b62d-4586-ac00-205ba64f6842","Type":"ContainerDied","Data":"189f182dafe23b1a9d87a956a8f7ec637d812c2134ca5739060178df78273dde"} Nov 03 19:07:43.163899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.163847 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8" Nov 03 19:07:43.170197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.170180 4948 scope.go:122] "RemoveContainer" containerID="b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640" Nov 03 19:07:43.170503 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:43.170476 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640\": container with ID starting with b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640 not found: ID does not exist" containerID="b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640" Nov 03 19:07:43.170585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.170504 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640"} err="failed to get container status \"b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640\": rpc error: code = NotFound desc = could not find container \"b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640\": container with ID starting with b136b1ceb9acb9e6518e8b704e324ab52ee5e650bd562cbcfe718551c49af640 not found: ID does not exist" Nov 03 19:07:43.170585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.170527 4948 scope.go:122] "RemoveContainer" containerID="107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3" Nov 03 19:07:43.195947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.195916 4948 scope.go:122] "RemoveContainer" containerID="107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3" Nov 03 19:07:43.196668 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:43.196608 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3\": container with ID starting with 107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3 not found: ID does not exist" containerID="107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3" Nov 03 19:07:43.196822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.196656 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3"} err="failed to get container status \"107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3\": rpc error: code = NotFound desc = could not find container \"107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3\": container with ID starting with 107865fc72addfd04a1f9630f589fdf1b24fecd00b0c4267ab3d53404b161bf3 not found: ID does not exist" Nov 03 19:07:43.196822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.196694 4948 scope.go:122] "RemoveContainer" containerID="9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9" Nov 03 19:07:43.199550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199522 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-credential-socket\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.199690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199598 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-certs\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.199690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199642 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a659e245-b62d-4586-ac00-205ba64f6842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a659e245-b62d-4586-ac00-205ba64f6842-istio-podinfo\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.199690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199735 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-data\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.199931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199825 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-kube-api-access-sz487\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-kube-api-access-sz487\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.199931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199862 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-socket\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.200046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199937 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-istio-token\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-istio-token\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.200046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.199974 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a659e245-b62d-4586-ac00-205ba64f6842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a659e245-b62d-4586-ac00-205ba64f6842-istiod-ca-cert\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.200149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200062 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-envoy\") pod \"a659e245-b62d-4586-ac00-205ba64f6842\" (UID: \"a659e245-b62d-4586-ac00-205ba64f6842\") " Nov 03 19:07:43.200149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200108 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-credential-socket" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:43.200149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200128 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-certs" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:43.200621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200411 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-socket" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:43.200621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200513 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a659e245-b62d-4586-ac00-205ba64f6842-istiod-ca-cert" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:43.200621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200596 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-data" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:43.200906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200882 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.200978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200912 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.200978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200927 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.200978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200942 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.200978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.200957 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a659e245-b62d-4586-ac00-205ba64f6842-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.202643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.202620 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-envoy" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:43.203184 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.203160 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a659e245-b62d-4586-ac00-205ba64f6842-istio-podinfo" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:43.203339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.203317 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-kube-api-access-sz487" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "kube-api-access-sz487". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:43.204302 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.204280 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-istio-token" pod "a659e245-b62d-4586-ac00-205ba64f6842" (UID: "a659e245-b62d-4586-ac00-205ba64f6842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:43.211221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.211133 4948 scope.go:122] "RemoveContainer" containerID="9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9" Nov 03 19:07:43.211489 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:43.211458 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9\": container with ID starting with 9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9 not found: ID does not exist" containerID="9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9" Nov 03 19:07:43.211593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.211505 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9"} err="failed to get container status \"9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9\": rpc error: code = NotFound desc = could not find container \"9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9\": container with ID starting with 9b650472d72bae1f1189b0c6e69cf26250b5f92450b4d7487295c807edfe73b9 not found: ID does not exist" Nov 03 19:07:43.211593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.211533 4948 scope.go:122] "RemoveContainer" containerID="bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2" Nov 03 19:07:43.227188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.227079 4948 scope.go:122] "RemoveContainer" containerID="bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2" Nov 03 19:07:43.227825 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:43.227794 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2\": container with ID starting with bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2 not found: ID does not exist" containerID="bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2" Nov 03 19:07:43.227940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.227829 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2"} err="failed to get container status \"bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2\": rpc error: code = NotFound desc = could not find container \"bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2\": container with ID starting with bfbd9ee7d37860a32a4cbc481c45da6486b1f129ccd4aa4bc3731119ba1b27a2 not found: ID does not exist" Nov 03 19:07:43.227940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.227854 4948 scope.go:122] "RemoveContainer" containerID="e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91" Nov 03 19:07:43.245441 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.245159 4948 scope.go:122] "RemoveContainer" containerID="e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91" Nov 03 19:07:43.245531 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:07:43.245489 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91\": container with ID starting with e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91 not found: ID does not exist" containerID="e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91" Nov 03 19:07:43.245570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.245524 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91"} err="failed to get container status \"e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91\": rpc error: code = NotFound desc = could not find container \"e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91\": container with ID starting with e817d400a30cfe1f0d6e20dc41d60ab008cd88c94edfec38f044fb03a8033a91 not found: ID does not exist" Nov 03 19:07:43.302369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.302328 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a659e245-b62d-4586-ac00-205ba64f6842-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.302369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.302362 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz487\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-kube-api-access-sz487\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.302559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.302380 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a659e245-b62d-4586-ac00-205ba64f6842-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.302559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.302392 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a659e245-b62d-4586-ac00-205ba64f6842-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.347235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.347196 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2"] Nov 03 19:07:43.352558 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.352526 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-6tlg2"] Nov 03 19:07:43.456756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.456719 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb"] Nov 03 19:07:43.461712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.461672 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-fwfsb"] Nov 03 19:07:43.573619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.573538 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh"] Nov 03 19:07:43.578163 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.578139 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-tpqgh"] Nov 03 19:07:43.821133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.821100 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6"] Nov 03 19:07:43.826060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.825980 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-jwhh6"] Nov 03 19:07:43.939587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.939558 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8"] Nov 03 19:07:43.957872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:43.957840 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-z4np8"] Nov 03 19:07:44.537722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.537031 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg"] Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538216 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538243 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538267 4948 cpu_manager.go:414] "containerMap: removing container" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538274 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538288 4948 cpu_manager.go:414] "containerMap: removing container" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538295 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538306 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538313 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538324 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538331 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538356 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e44b2cfb-1862-476f-9d40-259e440ba422" containerName="revoke-keys" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538363 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44b2cfb-1862-476f-9d40-259e440ba422" containerName="revoke-keys" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538378 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" containerName="payload-processing" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538385 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" containerName="payload-processing" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538402 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e3097574-7365-4487-9838-fe8fc220385b" containerName="payload-pre-processing" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538408 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3097574-7365-4487-9838-fe8fc220385b" containerName="payload-pre-processing" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538433 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538439 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538474 4948 cpu_manager.go:414] "containerMap: removing container" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538481 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538632 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3097574-7365-4487-9838-fe8fc220385b" containerName="payload-pre-processing" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538647 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a659e245-b62d-4586-ac00-205ba64f6842" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538659 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538672 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538682 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="d283713d-1736-4544-92a8-618ffc320af5" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538692 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538748 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20e356e-72df-49be-9213-ac45383f3950" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538760 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5837ad6-6036-4611-aa0d-149b7f52c356" containerName="istio-proxy" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538775 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44b2cfb-1862-476f-9d40-259e440ba422" containerName="revoke-keys" Nov 03 19:07:44.548722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.538786 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4375cc8-9a99-4fd7-a0d3-fead6d78cfa8" containerName="payload-processing" Nov 03 19:07:44.570829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.567986 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:44.604628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.595853 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg"] Nov 03 19:07:44.744402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.629075 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-bf3e1c-dockercfg-pmfd5\"" Nov 03 19:07:44.744402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.629179 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-bf3e1c\"" Nov 03 19:07:44.744402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.716842 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5715e-baf7-4c7a-bf28-b9440cb42e94-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:44.744402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.717373 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkxx\" (UniqueName: \"kubernetes.io/projected/25f5715e-baf7-4c7a-bf28-b9440cb42e94-kube-api-access-5vkxx\") pod \"payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:44.744402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.723366 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt"] Nov 03 19:07:44.828771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.819616 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkxx\" (UniqueName: \"kubernetes.io/projected/25f5715e-baf7-4c7a-bf28-b9440cb42e94-kube-api-access-5vkxx\") pod \"payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:44.828771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.819742 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5715e-baf7-4c7a-bf28-b9440cb42e94-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:44.828771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.820243 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:44.828771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.820875 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5715e-baf7-4c7a-bf28-b9440cb42e94-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:44.927731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.922304 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8559c03-65d3-4644-a58e-b63ef1cb475e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:44.927731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.922372 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjrs\" (UniqueName: \"kubernetes.io/projected/c8559c03-65d3-4644-a58e-b63ef1cb475e-kube-api-access-hgjrs\") pod \"payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:44.974336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.974109 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkxx\" (UniqueName: \"kubernetes.io/projected/25f5715e-baf7-4c7a-bf28-b9440cb42e94-kube-api-access-5vkxx\") pod \"payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:44.996026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.985891 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182a69ca-cec8-4640-82cd-04b021bea8e5" path="/var/lib/kubelet/pods/182a69ca-cec8-4640-82cd-04b021bea8e5/volumes" Nov 03 19:07:44.996026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.989282 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a659e245-b62d-4586-ac00-205ba64f6842" path="/var/lib/kubelet/pods/a659e245-b62d-4586-ac00-205ba64f6842/volumes" Nov 03 19:07:44.996026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.990060 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20e356e-72df-49be-9213-ac45383f3950" path="/var/lib/kubelet/pods/b20e356e-72df-49be-9213-ac45383f3950/volumes" Nov 03 19:07:44.996026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.990909 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d283713d-1736-4544-92a8-618ffc320af5" path="/var/lib/kubelet/pods/d283713d-1736-4544-92a8-618ffc320af5/volumes" Nov 03 19:07:44.996026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.991676 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3bed544-d72a-48d8-94fc-9c4f9707d16f" path="/var/lib/kubelet/pods/d3bed544-d72a-48d8-94fc-9c4f9707d16f/volumes" Nov 03 19:07:44.996026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:44.992328 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt"] Nov 03 19:07:45.024312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.024265 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8559c03-65d3-4644-a58e-b63ef1cb475e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:45.024467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.024331 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjrs\" (UniqueName: \"kubernetes.io/projected/c8559c03-65d3-4644-a58e-b63ef1cb475e-kube-api-access-hgjrs\") pod \"payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:45.025219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.025195 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8559c03-65d3-4644-a58e-b63ef1cb475e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:45.067117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.067081 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:45.087535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.087506 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjrs\" (UniqueName: \"kubernetes.io/projected/c8559c03-65d3-4644-a58e-b63ef1cb475e-kube-api-access-hgjrs\") pod \"payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:45.309269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.309239 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:45.456807 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:07:45.456764 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f5715e_baf7_4c7a_bf28_b9440cb42e94.slice/crio-cc26bd0284e3eb066920bd06658aa88261357aaee0e19a4a8305e168a1d205b2 WatchSource:0}: Error finding container cc26bd0284e3eb066920bd06658aa88261357aaee0e19a4a8305e168a1d205b2: Status 404 returned error can't find the container with id cc26bd0284e3eb066920bd06658aa88261357aaee0e19a4a8305e168a1d205b2 Nov 03 19:07:45.457460 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.457436 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg"] Nov 03 19:07:45.720259 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:07:45.720209 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8559c03_65d3_4644_a58e_b63ef1cb475e.slice/crio-a234bae995797e515ea434566402bfe67a7534643ba8c319d6902be0f6617e99 WatchSource:0}: Error finding container a234bae995797e515ea434566402bfe67a7534643ba8c319d6902be0f6617e99: Status 404 returned error can't find the container with id a234bae995797e515ea434566402bfe67a7534643ba8c319d6902be0f6617e99 Nov 03 19:07:45.745973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:45.745933 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt"] Nov 03 19:07:46.327790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:46.327748 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" event={"ID":"c8559c03-65d3-4644-a58e-b63ef1cb475e","Type":"ContainerStarted","Data":"935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4"} Nov 03 19:07:46.328273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:46.327822 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" event={"ID":"c8559c03-65d3-4644-a58e-b63ef1cb475e","Type":"ContainerStarted","Data":"a234bae995797e515ea434566402bfe67a7534643ba8c319d6902be0f6617e99"} Nov 03 19:07:46.328273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:46.327965 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:46.329603 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:46.329576 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" event={"ID":"25f5715e-baf7-4c7a-bf28-b9440cb42e94","Type":"ContainerStarted","Data":"d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45"} Nov 03 19:07:46.329747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:46.329623 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" event={"ID":"25f5715e-baf7-4c7a-bf28-b9440cb42e94","Type":"ContainerStarted","Data":"cc26bd0284e3eb066920bd06658aa88261357aaee0e19a4a8305e168a1d205b2"} Nov 03 19:07:46.329747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:46.329662 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:46.448892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:46.448840 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" podStartSLOduration=2.448826678 podStartE2EDuration="2.448826678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:46.445224923 +0000 UTC m=+1795.928220860" watchObservedRunningTime="2026-11-03 19:07:46.448826678 +0000 UTC m=+1795.931822592" Nov 03 19:07:51.850974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:51.850846 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" podStartSLOduration=7.850823528 podStartE2EDuration="7.850823528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:46.596048161 +0000 UTC m=+1796.079044077" watchObservedRunningTime="2026-11-03 19:07:51.850823528 +0000 UTC m=+1801.333819443" Nov 03 19:07:51.855337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:51.855306 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6"] Nov 03 19:07:51.904466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:51.904433 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6765e083-x88q6"] Nov 03 19:07:52.829727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:52.827722 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44b2cfb-1862-476f-9d40-259e440ba422" path="/var/lib/kubelet/pods/e44b2cfb-1862-476f-9d40-259e440ba422/volumes" Nov 03 19:07:53.721246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.721215 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5"] Nov 03 19:07:53.726038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.726015 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.780585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.780554 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ecf40a-openshift-default-dockercfg-j6m47\"" Nov 03 19:07:53.788954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.788918 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-data\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789010 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39803e8e-ba6f-449a-9d53-116d0147e05d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789061 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-credential-socket\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789090 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-envoy\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789147 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbnt\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-kube-api-access-gkbnt\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789176 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-socket\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789201 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-token\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789334 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-podinfo\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.789403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.789386 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-certs\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.837205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.837177 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5"] Nov 03 19:07:53.912727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.905564 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-credential-socket\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.912727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.905622 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-envoy\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.922991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.922929 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-credential-socket\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.924803 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbnt\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-kube-api-access-gkbnt\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.924848 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-socket\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.924872 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-token\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.924987 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-podinfo\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.925040 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-certs\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.925360 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-data\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.925438 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39803e8e-ba6f-449a-9d53-116d0147e05d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.926332 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39803e8e-ba6f-449a-9d53-116d0147e05d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.927750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.927175 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-socket\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.947256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.947188 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-certs\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:53.974098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:53.973932 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-data\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:54.151723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:54.108455 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-podinfo\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:54.195955 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:54.180818 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-token\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:54.195955 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:54.186525 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbnt\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-kube-api-access-gkbnt\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:54.214169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:54.214091 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-envoy\") pod \"e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:54.370776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:54.370329 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:55.058621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.058588 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5"] Nov 03 19:07:55.091490 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:07:55.091455 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39803e8e_ba6f_449a_9d53_116d0147e05d.slice/crio-a7d93fc2b4ee08963a8647fadd57ae282c78a824414c68db622303db13a6530b WatchSource:0}: Error finding container a7d93fc2b4ee08963a8647fadd57ae282c78a824414c68db622303db13a6530b: Status 404 returned error can't find the container with id a7d93fc2b4ee08963a8647fadd57ae282c78a824414c68db622303db13a6530b Nov 03 19:07:55.095016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.094977 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:55.095115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.095069 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:55.095152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.095114 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:07:55.404035 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.403954 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:07:55.411414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.411389 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:07:55.419813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.419791 4948 kubelet.go:1689] "Image garbage collection succeeded" Nov 03 19:07:55.476018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.475969 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" event={"ID":"39803e8e-ba6f-449a-9d53-116d0147e05d","Type":"ContainerStarted","Data":"8dc21d0b922a2231a1e793cae12e19ad49ad94ad2f0296117eb0cacd59b1b9db"} Nov 03 19:07:55.476210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.476086 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" event={"ID":"39803e8e-ba6f-449a-9d53-116d0147e05d","Type":"ContainerStarted","Data":"a7d93fc2b4ee08963a8647fadd57ae282c78a824414c68db622303db13a6530b"} Nov 03 19:07:55.609206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.609134 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podStartSLOduration=2.6091201870000003 podStartE2EDuration="2.609120187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:55.607788767 +0000 UTC m=+1805.090784682" watchObservedRunningTime="2026-11-03 19:07:55.609120187 +0000 UTC m=+1805.092116101" Nov 03 19:07:55.676504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.676421 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:07:55.686112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:55.686086 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:07:56.371841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:56.371805 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:56.378152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:56.378124 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:56.490525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:56.490484 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:56.491502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:56.491476 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:07:57.059435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:57.058930 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq"] Nov 03 19:07:57.066501 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:57.066462 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04333afe-7j4jq"] Nov 03 19:07:57.347806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:57.347688 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:07:57.352073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:57.349547 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:07:58.537481 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.537451 4948 scope.go:122] "RemoveContainer" containerID="afc71c9e5b4f75cfd50450aa771740254167c330fbb84d72b50b1b5df93c9df4" Nov 03 19:07:58.549170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.549149 4948 scope.go:122] "RemoveContainer" containerID="b214cc12af718b0001be62344b37ff9968663a5cf10151311156cda06e2da774" Nov 03 19:07:58.562189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.562126 4948 scope.go:122] "RemoveContainer" containerID="4d98e9f65eb171175c16eeb87f3990169dbef83a9955d59f5edbdf6fec597433" Nov 03 19:07:58.573166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.573147 4948 scope.go:122] "RemoveContainer" containerID="6fa43d726c1224f119130c7550532f461ad3074466647b4bc0af081447e1539c" Nov 03 19:07:58.583496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.583475 4948 scope.go:122] "RemoveContainer" containerID="7f4fe7d2291bffc5f8dca74e01b91b1363bda7f774a53ecc12f369e147c79109" Nov 03 19:07:58.593825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.593799 4948 scope.go:122] "RemoveContainer" containerID="e00520243afdf354a366881af60bc943acda0070591b36c3b8a3786c4f1a1480" Nov 03 19:07:58.604235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.604211 4948 scope.go:122] "RemoveContainer" containerID="af189019f9c484918d4d7b1c06b74fe961671f0271b6cf818b7b74335dfe7ccb" Nov 03 19:07:58.614685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.614662 4948 scope.go:122] "RemoveContainer" containerID="cbfce7620238b95876ef6e6dab9c9ec1ccef426c60bbd976077f489a28247ae5" Nov 03 19:07:58.624796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.624776 4948 scope.go:122] "RemoveContainer" containerID="f290352c51a53a7b699492748fc5ea9b8624ccc3c41944eb20b972825e411403" Nov 03 19:07:58.634609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.634592 4948 scope.go:122] "RemoveContainer" containerID="c2a0feef82a6166d90ce0965ce37f441fbe3ba4a3d8c9ded2ab8065c5783363c" Nov 03 19:07:58.644307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.644277 4948 scope.go:122] "RemoveContainer" containerID="5f388d941950caf50e2c5d6960b886e04555dbe2d95799cd661d59d221cf4839" Nov 03 19:07:58.658136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.658115 4948 scope.go:122] "RemoveContainer" containerID="77fa4fbc9001ea8edec8c86ccf36610520a56af453f36246688d078dabedb86c" Nov 03 19:07:58.668593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.668567 4948 scope.go:122] "RemoveContainer" containerID="4b4a2cff90fb14bd982b5a087723c0d1293b2c61d4e8440195a54ada5892dbca" Nov 03 19:07:58.681512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.681491 4948 scope.go:122] "RemoveContainer" containerID="7cfaf4e9803e8766f7657da78893a35a5c2eae3ee4ee4dd1a838bfdaee1f70e0" Nov 03 19:07:58.692380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.692358 4948 scope.go:122] "RemoveContainer" containerID="cadaf38fb50408f706b8719b9b0054fb2f8beafd5e02bd49e3bceca2f090bb46" Nov 03 19:07:58.768515 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:07:58.768487 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca36255-a69e-4c97-88a3-7af6fb78c099" path="/var/lib/kubelet/pods/aca36255-a69e-4c97-88a3-7af6fb78c099/volumes" Nov 03 19:08:02.438721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.428821 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722"] Nov 03 19:08:02.933536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.933508 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:02.947221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.947193 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg"] Nov 03 19:08:02.982034 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.981978 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722"] Nov 03 19:08:02.982206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.982054 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg"] Nov 03 19:08:02.982206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.982100 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9"] Nov 03 19:08:02.982396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.982371 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:02.986757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.986731 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9"] Nov 03 19:08:02.986890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.986844 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:02.991075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.991053 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ecf40a\"" Nov 03 19:08:02.992088 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:02.992064 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ecf40a-dockercfg-xqnws\"" Nov 03 19:08:02.995814 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:02.995792 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:08:03.011231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.011203 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpktw\" (UniqueName: \"kubernetes.io/projected/535fd0ff-312c-41d7-99f8-e04d5bafb029-kube-api-access-vpktw\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.011391 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.011376 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.011521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.011509 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.011573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.011559 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78788824-4241-4aec-b4c9-c677616c51c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:03.011670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.011659 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5nx\" (UniqueName: \"kubernetes.io/projected/4cedb0d2-901b-4225-840d-09fabaf00746-kube-api-access-xs5nx\") pod \"payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:03.011744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.011683 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnvw\" (UniqueName: \"kubernetes.io/projected/78788824-4241-4aec-b4c9-c677616c51c9-kube-api-access-4mnvw\") pod \"payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:03.011795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.011742 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cedb0d2-901b-4225-840d-09fabaf00746-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:03.091135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.091067 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:08:03.091135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.091072 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:08:03.091135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.091075 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:08:03.091135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.091119 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:08:03.113160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.113121 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5nx\" (UniqueName: \"kubernetes.io/projected/4cedb0d2-901b-4225-840d-09fabaf00746-kube-api-access-xs5nx\") pod \"payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:03.113342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.113179 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnvw\" (UniqueName: \"kubernetes.io/projected/78788824-4241-4aec-b4c9-c677616c51c9-kube-api-access-4mnvw\") pod \"payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:03.113342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.113236 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cedb0d2-901b-4225-840d-09fabaf00746-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:03.113342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.113272 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpktw\" (UniqueName: \"kubernetes.io/projected/535fd0ff-312c-41d7-99f8-e04d5bafb029-kube-api-access-vpktw\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.113541 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.113377 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.113541 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.113509 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.113661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.113573 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78788824-4241-4aec-b4c9-c677616c51c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:03.114234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.114210 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cedb0d2-901b-4225-840d-09fabaf00746-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:03.114518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.114493 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78788824-4241-4aec-b4c9-c677616c51c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:03.114838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.114815 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.119744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.116274 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.235949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.235630 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5nx\" (UniqueName: \"kubernetes.io/projected/4cedb0d2-901b-4225-840d-09fabaf00746-kube-api-access-xs5nx\") pod \"payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:03.236493 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.236417 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnvw\" (UniqueName: \"kubernetes.io/projected/78788824-4241-4aec-b4c9-c677616c51c9-kube-api-access-4mnvw\") pod \"payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:03.241768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.237291 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpktw\" (UniqueName: \"kubernetes.io/projected/535fd0ff-312c-41d7-99f8-e04d5bafb029-kube-api-access-vpktw\") pod \"maas-api-revoke-sub-095d4a9f-bm7hg\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.298232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.296768 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:03.314742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.313301 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:03.341769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.317120 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:03.922795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.921987 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g"] Nov 03 19:08:03.958812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.939097 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.032262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:03.984199 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-socket\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.073964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.073917 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-credential-socket\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.102708 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-podinfo\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.102939 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8xv\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-kube-api-access-pq8xv\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.103083 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-data\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.103157 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-certs\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.103236 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istiod-ca-cert\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.103444 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-envoy\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.103528 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-token\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.153520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.081541 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ba5674-openshift-default-dockercfg-dzll7\"" Nov 03 19:08:04.238723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.207097 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-certs\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.279488 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istiod-ca-cert\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.279747 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-envoy\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.279817 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-token\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.279899 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-socket\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.279990 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-credential-socket\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.280038 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-podinfo\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.280166 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8xv\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-kube-api-access-pq8xv\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.280269 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-data\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.239253 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-certs\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.294213 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istiod-ca-cert\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.296821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.294562 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-credential-socket\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.339882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.297817 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-data\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.339882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.325249 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-socket\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.379751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.358833 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g"] Nov 03 19:08:04.423832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.423797 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-podinfo\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.451723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.449957 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-token\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.463427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.463354 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8xv\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-kube-api-access-pq8xv\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.476633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.476596 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg"] Nov 03 19:08:04.501340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.500970 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-envoy\") pod \"e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.645794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.645760 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:04.692271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.688945 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9"] Nov 03 19:08:04.695762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:04.695500 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722"] Nov 03 19:08:05.045306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.045074 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" event={"ID":"78788824-4241-4aec-b4c9-c677616c51c9","Type":"ContainerStarted","Data":"1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec"} Nov 03 19:08:05.045306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.045188 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" event={"ID":"78788824-4241-4aec-b4c9-c677616c51c9","Type":"ContainerStarted","Data":"b2f6c9eadb2dca7a245bd079faf627bedcae2e12b6b6bacf9c7c254da6ba69d6"} Nov 03 19:08:05.045306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.045221 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:05.047594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.047558 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" event={"ID":"4cedb0d2-901b-4225-840d-09fabaf00746","Type":"ContainerStarted","Data":"352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38"} Nov 03 19:08:05.047771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.047615 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" event={"ID":"4cedb0d2-901b-4225-840d-09fabaf00746","Type":"ContainerStarted","Data":"210cd953a287102856975f27dd10fecb317caff31640cbc15f0fb9389526dfc5"} Nov 03 19:08:05.048474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.048452 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:05.065362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.065328 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" event={"ID":"535fd0ff-312c-41d7-99f8-e04d5bafb029","Type":"ContainerStarted","Data":"f70a8e89dc11e2174feb3fab577be7a133f05954cc6bcc3aea99a818f5bf0207"} Nov 03 19:08:05.066145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.066118 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" event={"ID":"535fd0ff-312c-41d7-99f8-e04d5bafb029","Type":"ContainerStarted","Data":"580db93f850195eeafaaa38b35799d10f42036f78d6cd4fe953baf423bbf4ac0"} Nov 03 19:08:05.090837 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:08:05.090802 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aa37b6_f6ca_4281_a298_e96c0f0417aa.slice/crio-5eb74aede5de079cea4611595b930a45dfc6828643c0bdc479f3a58ea49a7faf WatchSource:0}: Error finding container 5eb74aede5de079cea4611595b930a45dfc6828643c0bdc479f3a58ea49a7faf: Status 404 returned error can't find the container with id 5eb74aede5de079cea4611595b930a45dfc6828643c0bdc479f3a58ea49a7faf Nov 03 19:08:05.092261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.092237 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g"] Nov 03 19:08:05.093900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.093867 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:05.094002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.093945 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:05.094002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.093992 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:05.186809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.186756 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" podStartSLOduration=3.186736437 podStartE2EDuration="3.186736437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:05.185877927 +0000 UTC m=+1814.668873841" watchObservedRunningTime="2026-11-03 19:08:05.186736437 +0000 UTC m=+1814.669732352" Nov 03 19:08:05.333862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:05.333689 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" podStartSLOduration=3.333666916 podStartE2EDuration="3.333666916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:05.321890422 +0000 UTC m=+1814.804886337" watchObservedRunningTime="2026-11-03 19:08:05.333666916 +0000 UTC m=+1814.816662832" Nov 03 19:08:06.078344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.078248 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" event={"ID":"44aa37b6-f6ca-4281-a298-e96c0f0417aa","Type":"ContainerStarted","Data":"5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193"} Nov 03 19:08:06.078344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.078324 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" event={"ID":"44aa37b6-f6ca-4281-a298-e96c0f0417aa","Type":"ContainerStarted","Data":"5eb74aede5de079cea4611595b930a45dfc6828643c0bdc479f3a58ea49a7faf"} Nov 03 19:08:06.080649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.080622 4948 generic.go:356] "Generic (PLEG): container finished" podID="535fd0ff-312c-41d7-99f8-e04d5bafb029" containerID="f70a8e89dc11e2174feb3fab577be7a133f05954cc6bcc3aea99a818f5bf0207" exitCode=0 Nov 03 19:08:06.080808 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.080719 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" event={"ID":"535fd0ff-312c-41d7-99f8-e04d5bafb029","Type":"ContainerDied","Data":"f70a8e89dc11e2174feb3fab577be7a133f05954cc6bcc3aea99a818f5bf0207"} Nov 03 19:08:06.303976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.303910 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" podStartSLOduration=4.303889048 podStartE2EDuration="4.303889048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:05.503027461 +0000 UTC m=+1814.986023377" watchObservedRunningTime="2026-11-03 19:08:06.303889048 +0000 UTC m=+1815.786884963" Nov 03 19:08:06.308668 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.308617 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podStartSLOduration=3.308565021 podStartE2EDuration="3.308565021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:06.252359237 +0000 UTC m=+1815.735355152" watchObservedRunningTime="2026-11-03 19:08:06.308565021 +0000 UTC m=+1815.791560936" Nov 03 19:08:06.646586 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.646552 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:06.652569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:06.652541 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:07.119782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.119136 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:07.120272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.120252 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:08:07.447719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.445469 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:07.624653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.624615 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535fd0ff-312c-41d7-99f8-e04d5bafb029-kube-api-access-vpktw\" (UniqueName: \"kubernetes.io/projected/535fd0ff-312c-41d7-99f8-e04d5bafb029-kube-api-access-vpktw\") pod \"535fd0ff-312c-41d7-99f8-e04d5bafb029\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " Nov 03 19:08:07.625366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.625341 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-openshift-ca\") pod \"535fd0ff-312c-41d7-99f8-e04d5bafb029\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " Nov 03 19:08:07.625963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.625857 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-openshift-ca" pod "535fd0ff-312c-41d7-99f8-e04d5bafb029" (UID: "535fd0ff-312c-41d7-99f8-e04d5bafb029"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:07.626429 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.626409 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-xks-ca\") pod \"535fd0ff-312c-41d7-99f8-e04d5bafb029\" (UID: \"535fd0ff-312c-41d7-99f8-e04d5bafb029\") " Nov 03 19:08:07.627739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.627717 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:07.661168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.660382 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535fd0ff-312c-41d7-99f8-e04d5bafb029-kube-api-access-vpktw" pod "535fd0ff-312c-41d7-99f8-e04d5bafb029" (UID: "535fd0ff-312c-41d7-99f8-e04d5bafb029"). InnerVolumeSpecName "kube-api-access-vpktw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:07.663252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.663200 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-xks-ca" pod "535fd0ff-312c-41d7-99f8-e04d5bafb029" (UID: "535fd0ff-312c-41d7-99f8-e04d5bafb029"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:08:07.730063 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.729954 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535fd0ff-312c-41d7-99f8-e04d5bafb029-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:07.730063 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:07.729989 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpktw\" (UniqueName: \"kubernetes.io/projected/535fd0ff-312c-41d7-99f8-e04d5bafb029-kube-api-access-vpktw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:08.131544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:08.131462 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" Nov 03 19:08:08.131544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:08.131508 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg" event={"ID":"535fd0ff-312c-41d7-99f8-e04d5bafb029","Type":"ContainerDied","Data":"580db93f850195eeafaaa38b35799d10f42036f78d6cd4fe953baf423bbf4ac0"} Nov 03 19:08:08.132061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:08.131552 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="580db93f850195eeafaaa38b35799d10f42036f78d6cd4fe953baf423bbf4ac0" Nov 03 19:08:11.168795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:11.168754 4948 generic.go:356] "Generic (PLEG): container finished" podID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerID="08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d" exitCode=0 Nov 03 19:08:11.169353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:11.168801 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" event={"ID":"9a9a4423-52ad-4f7b-b92e-49d008755cbe","Type":"ContainerDied","Data":"08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d"} Nov 03 19:08:11.169420 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:11.169356 4948 scope.go:122] "RemoveContainer" containerID="08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d" Nov 03 19:08:11.171315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:11.171281 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:11.171423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:11.171356 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:11.171423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:11.171401 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:12.022722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.018102 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t"] Nov 03 19:08:12.022722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.019360 4948 cpu_manager.go:414] "containerMap: removing container" podUID="535fd0ff-312c-41d7-99f8-e04d5bafb029" containerName="revoke-keys" Nov 03 19:08:12.022722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.019382 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535fd0ff-312c-41d7-99f8-e04d5bafb029" containerName="revoke-keys" Nov 03 19:08:12.022722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.019646 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="535fd0ff-312c-41d7-99f8-e04d5bafb029" containerName="revoke-keys" Nov 03 19:08:12.032843 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.027616 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.125288 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.124726 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t"] Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.125951 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-certs\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126035 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-data\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126112 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-socket\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126173 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-envoy\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126259 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce38743-6104-4500-8cf0-0806409dce6e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126337 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce38743-6104-4500-8cf0-0806409dce6e-istio-podinfo\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126361 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-istio-token\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126388 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcmx\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-kube-api-access-5mcmx\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.128235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.126436 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-credential-socket\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.129546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.129520 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b485d8-openshift-default-dockercfg-w57lp\"" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234209 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-credential-socket\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234337 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-certs\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234438 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-data\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234546 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-socket\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234626 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-envoy\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234755 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce38743-6104-4500-8cf0-0806409dce6e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234854 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce38743-6104-4500-8cf0-0806409dce6e-istio-podinfo\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234882 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-istio-token\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.235725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.234909 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcmx\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-kube-api-access-5mcmx\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.304774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.257316 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-socket\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.304774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.257609 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-credential-socket\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.304774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.257879 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-certs\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.304774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.258111 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-data\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.304774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.258768 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce38743-6104-4500-8cf0-0806409dce6e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.403721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.403211 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-envoy\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.490722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.454723 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce38743-6104-4500-8cf0-0806409dce6e-istio-podinfo\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.490722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.486227 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-istio-token\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.616773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.612397 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcmx\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-kube-api-access-5mcmx\") pod \"e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:12.693728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.685987 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" event={"ID":"9a9a4423-52ad-4f7b-b92e-49d008755cbe","Type":"ContainerStarted","Data":"126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12"} Nov 03 19:08:12.744324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.742888 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:08:12.744532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.744338 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 19:08:12.744532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.744392 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 19:08:12.877115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:12.877012 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:13.348471 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:08:13.348438 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce38743_6104_4500_8cf0_0806409dce6e.slice/crio-08e7eed38c8016b6992006cab3e46e2c1d67c01517894ffbefd3c5839b8ec66c WatchSource:0}: Error finding container 08e7eed38c8016b6992006cab3e46e2c1d67c01517894ffbefd3c5839b8ec66c: Status 404 returned error can't find the container with id 08e7eed38c8016b6992006cab3e46e2c1d67c01517894ffbefd3c5839b8ec66c Nov 03 19:08:13.382905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.382849 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:13.388052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.388013 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:13.395753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.388292 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:13.396535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.384875 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t"] Nov 03 19:08:13.699628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.699593 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" event={"ID":"dce38743-6104-4500-8cf0-0806409dce6e","Type":"ContainerStarted","Data":"6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87"} Nov 03 19:08:13.699860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.699655 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" event={"ID":"dce38743-6104-4500-8cf0-0806409dce6e","Type":"ContainerStarted","Data":"08e7eed38c8016b6992006cab3e46e2c1d67c01517894ffbefd3c5839b8ec66c"} Nov 03 19:08:13.740765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.740734 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 19:08:13.740922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.740783 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 19:08:13.870724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.860329 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podStartSLOduration=2.860304266 podStartE2EDuration="2.860304266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:13.843042315 +0000 UTC m=+1823.326038230" watchObservedRunningTime="2026-11-03 19:08:13.860304266 +0000 UTC m=+1823.343300181" Nov 03 19:08:13.915725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:13.878791 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:14.111321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:14.111231 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:08:15.051016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.034072 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 19:08:15.051016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.034126 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 19:08:15.398327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.377432 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:15.398327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.377500 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs"] Nov 03 19:08:15.568482 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.568451 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:15.568830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.568813 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:08:15.569002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.568987 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs"] Nov 03 19:08:15.569242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.569221 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:15.653765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.653637 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424"] Nov 03 19:08:15.704825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.704762 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:08:15.747017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.710179 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:15.804419 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.746494 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c89c35e-a409-4f07-a68d-31884163253b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:15.804865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.804778 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8008ec73-e7ca-4700-995b-2978b89688b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:15.805080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.748690 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 19:08:15.806571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.805276 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 19:08:15.806571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.746731 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ba5674\"" Nov 03 19:08:15.806571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.793754 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424"] Nov 03 19:08:15.806571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.805368 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqld7\" (UniqueName: \"kubernetes.io/projected/4c89c35e-a409-4f07-a68d-31884163253b-kube-api-access-qqld7\") pod \"payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:15.806571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.805429 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlwt\" (UniqueName: \"kubernetes.io/projected/8008ec73-e7ca-4700-995b-2978b89688b2-kube-api-access-jhlwt\") pod \"payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:15.877096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.877057 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ba5674-dockercfg-xc9g2\"" Nov 03 19:08:15.907532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.907447 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c89c35e-a409-4f07-a68d-31884163253b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:15.907532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.907498 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8008ec73-e7ca-4700-995b-2978b89688b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:15.907746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.907635 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqld7\" (UniqueName: \"kubernetes.io/projected/4c89c35e-a409-4f07-a68d-31884163253b-kube-api-access-qqld7\") pod \"payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:15.907746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.907673 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlwt\" (UniqueName: \"kubernetes.io/projected/8008ec73-e7ca-4700-995b-2978b89688b2-kube-api-access-jhlwt\") pod \"payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:15.908395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.908365 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c89c35e-a409-4f07-a68d-31884163253b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:15.908778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.908761 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8008ec73-e7ca-4700-995b-2978b89688b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:15.991527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.991489 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlwt\" (UniqueName: \"kubernetes.io/projected/8008ec73-e7ca-4700-995b-2978b89688b2-kube-api-access-jhlwt\") pod \"payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:15.995640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:15.995607 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqld7\" (UniqueName: \"kubernetes.io/projected/4c89c35e-a409-4f07-a68d-31884163253b-kube-api-access-qqld7\") pod \"payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:16.088435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:16.088405 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:08:16.114551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:16.114523 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:16.122877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:16.122852 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:16.631988 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:08:16.631948 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c89c35e_a409_4f07_a68d_31884163253b.slice/crio-efb4ecfb3232d8c90bffa0746d37e6c8a6af3f79bbcfa100ba47488ace749b16 WatchSource:0}: Error finding container efb4ecfb3232d8c90bffa0746d37e6c8a6af3f79bbcfa100ba47488ace749b16: Status 404 returned error can't find the container with id efb4ecfb3232d8c90bffa0746d37e6c8a6af3f79bbcfa100ba47488ace749b16 Nov 03 19:08:16.632809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:16.632784 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs"] Nov 03 19:08:16.703101 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:08:16.703071 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8008ec73_e7ca_4700_995b_2978b89688b2.slice/crio-22264636b0b63862b8013c3ab6c2fa8a35d463e3f52324b523e0dad2d5261833 WatchSource:0}: Error finding container 22264636b0b63862b8013c3ab6c2fa8a35d463e3f52324b523e0dad2d5261833: Status 404 returned error can't find the container with id 22264636b0b63862b8013c3ab6c2fa8a35d463e3f52324b523e0dad2d5261833 Nov 03 19:08:16.704966 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:16.704941 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424"] Nov 03 19:08:16.740297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:16.740049 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 19:08:16.740297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:16.740136 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 19:08:17.120950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.120919 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:08:17.411608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.411514 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" event={"ID":"8008ec73-e7ca-4700-995b-2978b89688b2","Type":"ContainerStarted","Data":"9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f"} Nov 03 19:08:17.411820 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.411657 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" event={"ID":"8008ec73-e7ca-4700-995b-2978b89688b2","Type":"ContainerStarted","Data":"22264636b0b63862b8013c3ab6c2fa8a35d463e3f52324b523e0dad2d5261833"} Nov 03 19:08:17.411820 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.411692 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:17.414235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.414174 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" event={"ID":"4c89c35e-a409-4f07-a68d-31884163253b","Type":"ContainerStarted","Data":"3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61"} Nov 03 19:08:17.414235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.414229 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" event={"ID":"4c89c35e-a409-4f07-a68d-31884163253b","Type":"ContainerStarted","Data":"efb4ecfb3232d8c90bffa0746d37e6c8a6af3f79bbcfa100ba47488ace749b16"} Nov 03 19:08:17.414416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.414325 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:17.575981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.575904 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" podStartSLOduration=2.57588017 podStartE2EDuration="2.57588017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:17.550587508 +0000 UTC m=+1827.033583422" watchObservedRunningTime="2026-11-03 19:08:17.57588017 +0000 UTC m=+1827.058876087" Nov 03 19:08:17.808726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.808258 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 19:08:17.808726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.808319 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 19:08:17.841957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:17.823046 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" podStartSLOduration=3.823024384 podStartE2EDuration="3.823024384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:17.680482455 +0000 UTC m=+1827.163478371" watchObservedRunningTime="2026-11-03 19:08:17.823024384 +0000 UTC m=+1827.306020299" Nov 03 19:08:18.739813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:18.739773 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 19:08:18.740225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:18.739850 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 19:08:19.745224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:19.745194 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:08:19.746229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:19.746206 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:08:20.908644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:20.908597 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl"] Nov 03 19:08:20.930487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:20.930446 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl"] Nov 03 19:08:20.934198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:20.933615 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:20.987191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:20.977513 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht"] Nov 03 19:08:21.071722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.071338 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b485d8\"" Nov 03 19:08:21.074721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.072685 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b485d8-dockercfg-xrbm2\"" Nov 03 19:08:21.110725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.105741 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d087b8-82fd-41c7-b0a7-786552d2b041-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:21.110725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.105854 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqpr\" (UniqueName: \"kubernetes.io/projected/14d087b8-82fd-41c7-b0a7-786552d2b041-kube-api-access-xfqpr\") pod \"payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:21.253722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.253438 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d087b8-82fd-41c7-b0a7-786552d2b041-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:21.253722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.253522 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqpr\" (UniqueName: \"kubernetes.io/projected/14d087b8-82fd-41c7-b0a7-786552d2b041-kube-api-access-xfqpr\") pod \"payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:21.259328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.254999 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d087b8-82fd-41c7-b0a7-786552d2b041-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:21.319908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.319844 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht"] Nov 03 19:08:21.320353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.320329 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:21.371801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.371763 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqpr\" (UniqueName: \"kubernetes.io/projected/14d087b8-82fd-41c7-b0a7-786552d2b041-kube-api-access-xfqpr\") pod \"payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:21.530792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.520258 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvr66\" (UniqueName: \"kubernetes.io/projected/18dd560f-7ce0-46ef-a13a-895c10d116d3-kube-api-access-xvr66\") pod \"payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:21.530792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.520572 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd560f-7ce0-46ef-a13a-895c10d116d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:21.625323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.623546 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd560f-7ce0-46ef-a13a-895c10d116d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:21.625323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.624975 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvr66\" (UniqueName: \"kubernetes.io/projected/18dd560f-7ce0-46ef-a13a-895c10d116d3-kube-api-access-xvr66\") pod \"payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:21.627078 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.626414 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd560f-7ce0-46ef-a13a-895c10d116d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:21.640471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.640440 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:21.689154 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.689067 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvr66\" (UniqueName: \"kubernetes.io/projected/18dd560f-7ce0-46ef-a13a-895c10d116d3-kube-api-access-xvr66\") pod \"payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:21.914841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:21.914767 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:22.033546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.033496 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl"] Nov 03 19:08:22.299468 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:08:22.299433 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dd560f_7ce0_46ef_a13a_895c10d116d3.slice/crio-e56e7f462a3018c006b90cadab3b6bade1a834d1ad66bc72d7697a7cebead85f WatchSource:0}: Error finding container e56e7f462a3018c006b90cadab3b6bade1a834d1ad66bc72d7697a7cebead85f: Status 404 returned error can't find the container with id e56e7f462a3018c006b90cadab3b6bade1a834d1ad66bc72d7697a7cebead85f Nov 03 19:08:22.300737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.300687 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht"] Nov 03 19:08:22.862858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.859127 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d"] Nov 03 19:08:22.916730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.916079 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" event={"ID":"18dd560f-7ce0-46ef-a13a-895c10d116d3","Type":"ContainerStarted","Data":"6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf"} Nov 03 19:08:22.916730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.916161 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" event={"ID":"18dd560f-7ce0-46ef-a13a-895c10d116d3","Type":"ContainerStarted","Data":"e56e7f462a3018c006b90cadab3b6bade1a834d1ad66bc72d7697a7cebead85f"} Nov 03 19:08:22.917265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.916751 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:22.958174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.955670 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" containerID="cri-o://7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3" gracePeriod=30 Nov 03 19:08:22.977735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.962846 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" event={"ID":"14d087b8-82fd-41c7-b0a7-786552d2b041","Type":"ContainerStarted","Data":"0291c2b359ffe391cfaf31a18d6388fe5d5a4b51b6a843bf1268ebe8523464c4"} Nov 03 19:08:23.004746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.978032 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" event={"ID":"14d087b8-82fd-41c7-b0a7-786552d2b041","Type":"ContainerStarted","Data":"0e74834ef1960128e081b4465f0e0906af4abd762b9947a06c8e6245552bed7f"} Nov 03 19:08:23.004746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:22.978093 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:23.314728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:23.313479 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" podStartSLOduration=3.313459528 podStartE2EDuration="3.313459528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:23.266386117 +0000 UTC m=+1832.749382035" watchObservedRunningTime="2026-11-03 19:08:23.313459528 +0000 UTC m=+1832.796455445" Nov 03 19:08:23.518730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:23.511081 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" podStartSLOduration=3.511056496 podStartE2EDuration="3.511056496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:23.390935773 +0000 UTC m=+1832.873931690" watchObservedRunningTime="2026-11-03 19:08:23.511056496 +0000 UTC m=+1832.994052413" Nov 03 19:08:26.047725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:26.046898 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:26.094929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:26.094882 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:26.279722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:26.277803 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:26.370475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:26.370361 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:26.420372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:26.420329 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:26.670897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:26.670787 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:27.193725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.192774 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:08:27.267724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.262552 4948 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:27.371003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.370960 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:27.371787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.371764 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:08:27.420957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.420910 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:27.421800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.421776 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:08:27.710720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.706386 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 19:08:27.719991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.718127 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:08:27.744790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.744763 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:08:27.823466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.823431 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:08:27.824443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:27.824419 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:08:28.428471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.428437 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:08:28.428950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.428531 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:08:28.618986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.618957 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:08:28.720320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720267 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-podinfo\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.720503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720351 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-token\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-token\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.720503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720415 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651bbe89-2f02-462c-bc24-cf5a69fbd101-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651bbe89-2f02-462c-bc24-cf5a69fbd101-istiod-ca-cert\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.720634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720601 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-envoy\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.720709 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720636 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-certs\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.720790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720768 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-kube-api-access-vz7dq\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-kube-api-access-vz7dq\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.720872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720857 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-data\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.720986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.720971 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-socket\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.721058 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.721044 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-credential-socket\") pod \"651bbe89-2f02-462c-bc24-cf5a69fbd101\" (UID: \"651bbe89-2f02-462c-bc24-cf5a69fbd101\") " Nov 03 19:08:28.722791 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.722734 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651bbe89-2f02-462c-bc24-cf5a69fbd101-istiod-ca-cert" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:28.724888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.724855 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-credential-socket" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:28.725098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.725076 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-data" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:28.725191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.725176 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-socket" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:28.725330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.725310 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-certs" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:28.726013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.725976 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-podinfo" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:28.726314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.726286 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-envoy" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:28.727218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.727188 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-kube-api-access-vz7dq" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "kube-api-access-vz7dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:28.727937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.727908 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-token" pod "651bbe89-2f02-462c-bc24-cf5a69fbd101" (UID: "651bbe89-2f02-462c-bc24-cf5a69fbd101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:28.824761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824657 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.824761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824740 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.824761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824758 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651bbe89-2f02-462c-bc24-cf5a69fbd101-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.825052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824772 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.825052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824786 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.825052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824800 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz7dq\" (UniqueName: \"kubernetes.io/projected/651bbe89-2f02-462c-bc24-cf5a69fbd101-kube-api-access-vz7dq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.825052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824814 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.825052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824882 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:28.825052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:28.824899 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651bbe89-2f02-462c-bc24-cf5a69fbd101-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:29.342438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.341585 4948 generic.go:356] "Generic (PLEG): container finished" podID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerID="7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3" exitCode=0 Nov 03 19:08:29.342438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.341658 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" Nov 03 19:08:29.342438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.341878 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" event={"ID":"651bbe89-2f02-462c-bc24-cf5a69fbd101","Type":"ContainerDied","Data":"7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3"} Nov 03 19:08:29.342438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.341954 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" event={"ID":"651bbe89-2f02-462c-bc24-cf5a69fbd101","Type":"ContainerDied","Data":"b150d107528b1636e14e8d7c09e23b26f8394f707d29ee2e067fd9707bd7582a"} Nov 03 19:08:29.342438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.341985 4948 scope.go:122] "RemoveContainer" containerID="7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3" Nov 03 19:08:29.383788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.383750 4948 scope.go:122] "RemoveContainer" containerID="7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3" Nov 03 19:08:29.384190 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:29.384160 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3\": container with ID starting with 7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3 not found: ID does not exist" containerID="7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3" Nov 03 19:08:29.384248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.384205 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3"} err="failed to get container status \"7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3\": rpc error: code = NotFound desc = could not find container \"7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3\": container with ID starting with 7dbabbed212c3bd98c251f7110dd0ac919051b6a91587cab29ac62fd0f4916b3 not found: ID does not exist" Nov 03 19:08:29.516951 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.516918 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d"] Nov 03 19:08:29.525526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.525493 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:29.525642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.525560 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:29.578662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:29.578624 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-z9f5d"] Nov 03 19:08:30.784695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:30.784617 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" path="/var/lib/kubelet/pods/651bbe89-2f02-462c-bc24-cf5a69fbd101/volumes" Nov 03 19:08:32.804891 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:32.804852 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:08:33.976171 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:33.976132 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:08:33.976569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:33.976210 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:08:34.077866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:34.077825 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:34.078059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:34.077886 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:35.444706 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:35.444673 4948 generic.go:356] "Generic (PLEG): container finished" podID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerID="8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641" exitCode=0 Nov 03 19:08:35.445146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:35.444749 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" event={"ID":"b99a5d5a-0550-45cf-8806-1daeb44e1730","Type":"ContainerDied","Data":"8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641"} Nov 03 19:08:35.445274 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:35.445246 4948 scope.go:122] "RemoveContainer" containerID="8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641" Nov 03 19:08:35.447083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:35.447050 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:35.447225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:35.447138 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:35.447225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:35.447188 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:08:36.457619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:36.457580 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" event={"ID":"b99a5d5a-0550-45cf-8806-1daeb44e1730","Type":"ContainerStarted","Data":"679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e"} Nov 03 19:08:37.050001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:37.049970 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:08:37.050220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:37.050133 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 19:08:37.050220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:37.050203 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 19:08:38.049891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.049857 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 19:08:38.050319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.049922 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 19:08:38.422587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.422500 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs"] Nov 03 19:08:38.422779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.422749 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" containerID="cri-o://def6eb8ca2e56203760b60e71e6ed5904487bb785fd0025445645e15f820a207" gracePeriod=30 Nov 03 19:08:38.492188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.492136 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j"] Nov 03 19:08:38.492426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.492399 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" containerID="cri-o://86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a" gracePeriod=30 Nov 03 19:08:38.503563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.503523 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6"] Nov 03 19:08:38.504073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:38.504003 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" containerID="cri-o://5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765" gracePeriod=30 Nov 03 19:08:39.049573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:39.049538 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 19:08:39.049797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:39.049599 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 19:08:40.049552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:40.049512 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 19:08:40.050067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:40.049580 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 19:08:41.069447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:41.052158 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 19:08:41.070253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:41.070211 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 19:08:41.289377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:41.289338 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:08:41.289657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:41.289631 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:08:42.049779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:42.049742 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 19:08:42.049972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:42.049816 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 19:08:43.054876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.054835 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:08:43.055602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.055579 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:08:43.056221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.056199 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:08:43.491306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.491265 4948 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:08:43.491488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.491339 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 19:08:43.562172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.562144 4948 generic.go:356] "Generic (PLEG): container finished" podID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerID="def6eb8ca2e56203760b60e71e6ed5904487bb785fd0025445645e15f820a207" exitCode=0 Nov 03 19:08:43.562278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.562217 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" event={"ID":"9b1e7095-d0f7-4548-ae01-1ea485ece334","Type":"ContainerDied","Data":"def6eb8ca2e56203760b60e71e6ed5904487bb785fd0025445645e15f820a207"} Nov 03 19:08:43.848117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.848089 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:08:43.859665 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.859561 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:08:43.941932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.941877 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:08:43.946139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946110 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-podinfo\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.946267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946213 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-socket\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.946267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946258 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1e7095-d0f7-4548-ae01-1ea485ece334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1e7095-d0f7-4548-ae01-1ea485ece334-istiod-ca-cert\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.946358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946308 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-socket\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.946414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946358 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-data\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.946489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946435 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istiod-ca-cert\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.946552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946478 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-socket" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.946608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946590 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-envoy\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.946670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946625 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-credential-socket\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.946670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946640 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-socket" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.946670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946638 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-data" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.946670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946659 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-credential-socket\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.946940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946798 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-certs\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.946940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946869 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1e7095-d0f7-4548-ae01-1ea485ece334-istiod-ca-cert" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:43.946940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946900 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-certs\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.947101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946932 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-credential-socket" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.947101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946950 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-token\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-token\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.947101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946966 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istiod-ca-cert" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:43.947101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.946979 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-kube-api-access-nq4bl\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-kube-api-access-nq4bl\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.947101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947066 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-podinfo\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.947371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947106 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-token\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-token\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.947371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947090 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-certs" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.947371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947153 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-certs" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.947371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947278 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-kube-api-access-zgc4f\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-kube-api-access-zgc4f\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.947371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947355 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-credential-socket" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.947654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947403 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-data\") pod \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\" (UID: \"8e4b1269-cfed-484b-8957-d8eb9b3dbb43\") " Nov 03 19:08:43.947654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947445 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-envoy\") pod \"9b1e7095-d0f7-4548-ae01-1ea485ece334\" (UID: \"9b1e7095-d0f7-4548-ae01-1ea485ece334\") " Nov 03 19:08:43.947971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.947941 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-data" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.948400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948372 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1e7095-d0f7-4548-ae01-1ea485ece334-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948404 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948423 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948436 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948452 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948466 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948479 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948496 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948510 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948525 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:43.948910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.948889 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-podinfo" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:43.949584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.949560 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-kube-api-access-nq4bl" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "kube-api-access-nq4bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:43.949681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.949597 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-token" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:43.949681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.949629 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-podinfo" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:43.949884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.949862 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-kube-api-access-zgc4f" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "kube-api-access-zgc4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:43.950396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.950375 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-envoy" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.950687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.950672 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-envoy" pod "8e4b1269-cfed-484b-8957-d8eb9b3dbb43" (UID: "8e4b1269-cfed-484b-8957-d8eb9b3dbb43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:43.951202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:43.951182 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-token" pod "9b1e7095-d0f7-4548-ae01-1ea485ece334" (UID: "9b1e7095-d0f7-4548-ae01-1ea485ece334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.050621 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-podinfo\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.050753 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-envoy\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.050856 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-kube-api-access-66h9v\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-kube-api-access-66h9v\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.050912 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-token\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-token\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.050952 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-data\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.051039 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-socket\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.051181 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b567e2b-d032-4733-a6bb-6531f0d149a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b567e2b-d032-4733-a6bb-6531f0d149a2-istiod-ca-cert\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.051212 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-certs\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.051313 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-credential-socket\") pod \"7b567e2b-d032-4733-a6bb-6531f0d149a2\" (UID: \"7b567e2b-d032-4733-a6bb-6531f0d149a2\") " Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052760 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052793 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052809 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052824 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052902 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq4bl\" (UniqueName: \"kubernetes.io/projected/8e4b1269-cfed-484b-8957-d8eb9b3dbb43-kube-api-access-nq4bl\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052927 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052965 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.052981 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgc4f\" (UniqueName: \"kubernetes.io/projected/9b1e7095-d0f7-4548-ae01-1ea485ece334-kube-api-access-zgc4f\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.053271 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-credential-socket" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.054223 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-data" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.054395 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-socket" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.054678 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b567e2b-d032-4733-a6bb-6531f0d149a2-istiod-ca-cert" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:44.055716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.054936 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-certs" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:44.063089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.062991 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-envoy" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:44.064824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.064787 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-token" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:44.066361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.066332 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-podinfo" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:44.073577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.073544 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-kube-api-access-66h9v" pod "7b567e2b-d032-4733-a6bb-6531f0d149a2" (UID: "7b567e2b-d032-4733-a6bb-6531f0d149a2"). InnerVolumeSpecName "kube-api-access-66h9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:44.155043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.154994 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155034 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b567e2b-d032-4733-a6bb-6531f0d149a2-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155050 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155066 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155080 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155093 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155107 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66h9v\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-kube-api-access-66h9v\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155122 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.155301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.155135 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b567e2b-d032-4733-a6bb-6531f0d149a2-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:44.574065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.574026 4948 generic.go:356] "Generic (PLEG): container finished" podID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerID="86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a" exitCode=0 Nov 03 19:08:44.574257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.574090 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" Nov 03 19:08:44.574257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.574116 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" event={"ID":"8e4b1269-cfed-484b-8957-d8eb9b3dbb43","Type":"ContainerDied","Data":"86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a"} Nov 03 19:08:44.574257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.574217 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j" event={"ID":"8e4b1269-cfed-484b-8957-d8eb9b3dbb43","Type":"ContainerDied","Data":"d218690925b3c5de637904d15f009f329369a95eec16fb2e8d3c84a60213e8a8"} Nov 03 19:08:44.574257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.574249 4948 scope.go:122] "RemoveContainer" containerID="86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a" Nov 03 19:08:44.576068 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.576047 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" Nov 03 19:08:44.576343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.576074 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs" event={"ID":"9b1e7095-d0f7-4548-ae01-1ea485ece334","Type":"ContainerDied","Data":"65e06d8340bc1502b212ea07df101d0767ed9fa4e6c25289ba18dcf5ea7b2b57"} Nov 03 19:08:44.578227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.578200 4948 generic.go:356] "Generic (PLEG): container finished" podID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerID="5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765" exitCode=0 Nov 03 19:08:44.578362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.578292 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" Nov 03 19:08:44.578436 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.578363 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" event={"ID":"7b567e2b-d032-4733-a6bb-6531f0d149a2","Type":"ContainerDied","Data":"5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765"} Nov 03 19:08:44.578436 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.578405 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6" event={"ID":"7b567e2b-d032-4733-a6bb-6531f0d149a2","Type":"ContainerDied","Data":"9f6f23bfa09333b79104ac8111578b62fdd1afec7a39c5b150efa472964281ec"} Nov 03 19:08:44.592112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.592091 4948 scope.go:122] "RemoveContainer" containerID="86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a" Nov 03 19:08:44.592509 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:44.592483 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a\": container with ID starting with 86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a not found: ID does not exist" containerID="86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a" Nov 03 19:08:44.592584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.592520 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a"} err="failed to get container status \"86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a\": rpc error: code = NotFound desc = could not find container \"86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a\": container with ID starting with 86aea0843e683381952cd13842b21752b9f5a7d019168d1462a0dca87c8a992a not found: ID does not exist" Nov 03 19:08:44.592584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.592546 4948 scope.go:122] "RemoveContainer" containerID="def6eb8ca2e56203760b60e71e6ed5904487bb785fd0025445645e15f820a207" Nov 03 19:08:44.606046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.606025 4948 scope.go:122] "RemoveContainer" containerID="5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765" Nov 03 19:08:44.617138 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.616982 4948 scope.go:122] "RemoveContainer" containerID="5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765" Nov 03 19:08:44.617284 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:44.617259 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765\": container with ID starting with 5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765 not found: ID does not exist" containerID="5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765" Nov 03 19:08:44.617332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.617297 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765"} err="failed to get container status \"5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765\": rpc error: code = NotFound desc = could not find container \"5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765\": container with ID starting with 5834b5434db34bbd7ba0c25f82649ea0806a9925c27ea4ebcefb2d4aa69d8765 not found: ID does not exist" Nov 03 19:08:44.745426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.745348 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j"] Nov 03 19:08:44.799248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.799209 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hrs6j"] Nov 03 19:08:44.866546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.866515 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs"] Nov 03 19:08:44.874489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.874457 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-2t6vs"] Nov 03 19:08:44.983554 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.983519 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6"] Nov 03 19:08:44.991170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:44.991141 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-9kbd6"] Nov 03 19:08:46.759409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:46.759370 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" path="/var/lib/kubelet/pods/7b567e2b-d032-4733-a6bb-6531f0d149a2/volumes" Nov 03 19:08:46.760048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:46.760030 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" path="/var/lib/kubelet/pods/8e4b1269-cfed-484b-8957-d8eb9b3dbb43/volumes" Nov 03 19:08:46.760564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:46.760545 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" path="/var/lib/kubelet/pods/9b1e7095-d0f7-4548-ae01-1ea485ece334/volumes" Nov 03 19:08:49.242194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:49.241411 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:49.763305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:49.762081 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:50.202224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:50.200805 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:50.243914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:50.241641 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:50.243914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:50.242229 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:08:50.260655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:50.260098 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:50.508946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:50.508861 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:08:52.031421 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.031391 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz"] Nov 03 19:08:52.031923 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.031572 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" containerID="cri-o://a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda" gracePeriod=30 Nov 03 19:08:52.082293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.082259 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt"] Nov 03 19:08:52.082489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.082468 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" containerID="cri-o://0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d" gracePeriod=30 Nov 03 19:08:52.088096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.088067 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5"] Nov 03 19:08:52.088349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.088323 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" containerID="cri-o://6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8" gracePeriod=30 Nov 03 19:08:52.143201 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.143162 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw"] Nov 03 19:08:52.143750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.143690 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" containerID="cri-o://b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd" gracePeriod=30 Nov 03 19:08:52.230272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.230231 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz"] Nov 03 19:08:52.230590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.230538 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" containerID="cri-o://70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b" gracePeriod=30 Nov 03 19:08:52.242995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.239584 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx"] Nov 03 19:08:52.242995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.239882 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" containerID="cri-o://f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144" gracePeriod=30 Nov 03 19:08:52.257660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.257044 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv"] Nov 03 19:08:52.257660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.257284 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5"] Nov 03 19:08:52.257660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.257279 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" containerID="cri-o://31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510" gracePeriod=30 Nov 03 19:08:52.257660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.257484 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" containerID="cri-o://7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630" gracePeriod=30 Nov 03 19:08:52.486479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.486440 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:08:52.486662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:52.486510 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:08:54.058444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:54.058407 4948 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:08:54.058925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:54.058480 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:08:54.641039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:54.641003 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:08:54.641216 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:54.641064 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:08:55.922313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:55.922280 4948 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:08:55.922786 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:55.922340 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:08:57.528567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.528521 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:08:57.588935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.588905 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:08:57.592187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592158 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-kube-api-access-29swq\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-kube-api-access-29swq\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.592311 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592233 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-certs\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.592311 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592298 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-socket\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.592403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592387 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-credential-socket\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.592449 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592428 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-envoy\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.592487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592459 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-token\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-token\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.592525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592504 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-kube-api-access-949r8\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-kube-api-access-949r8\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.592557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592537 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-data\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.592606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592583 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29a321b-415d-413f-9d6e-d38150ef8e67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29a321b-415d-413f-9d6e-d38150ef8e67-istiod-ca-cert\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.592656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592628 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-podinfo\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.592740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592724 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-credential-socket\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.592824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592809 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-token\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-token\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.592907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.592889 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-socket\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.593021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.593002 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-certs\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.593743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.593122 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-podinfo\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.593743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.593162 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-data\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.593743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.593201 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istiod-ca-cert\") pod \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\" (UID: \"cd15b0fc-c46e-438c-a583-0e69d0dacb5d\") " Nov 03 19:08:57.593743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.593234 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-envoy\") pod \"f29a321b-415d-413f-9d6e-d38150ef8e67\" (UID: \"f29a321b-415d-413f-9d6e-d38150ef8e67\") " Nov 03 19:08:57.595201 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.595157 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-certs" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.596743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.596154 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-certs" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.596743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.596293 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-data" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.596743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.596375 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-credential-socket" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.596743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.596400 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-socket" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.597563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.597499 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istiod-ca-cert" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.600822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.600796 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-socket" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.601616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.601594 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-data" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.601914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.601889 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-credential-socket" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.602004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.601959 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29a321b-415d-413f-9d6e-d38150ef8e67-istiod-ca-cert" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.602289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.602261 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-envoy" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.602812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.602755 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-podinfo" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.602812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.602789 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-token" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.602962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.602824 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-kube-api-access-949r8" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "kube-api-access-949r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.603345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.603317 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-envoy" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.605324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.604007 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-podinfo" pod "f29a321b-415d-413f-9d6e-d38150ef8e67" (UID: "f29a321b-415d-413f-9d6e-d38150ef8e67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.605324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.604104 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-kube-api-access-29swq" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "kube-api-access-29swq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.605637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.605605 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-token" pod "cd15b0fc-c46e-438c-a583-0e69d0dacb5d" (UID: "cd15b0fc-c46e-438c-a583-0e69d0dacb5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.653615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.652787 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 19:08:57.695889 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.695580 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-token\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-token\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.696474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.695924 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-socket\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.696582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.696555 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-socket" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.696693 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.696670 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-certs\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.696943 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-certs" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.697400 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-data" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.697439 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-data\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.697517 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-credential-socket\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.697549 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-envoy\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.697623 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-podinfo\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.697804 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istiod-ca-cert\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.698071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.697848 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-kube-api-access-rrclq\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-kube-api-access-rrclq\") pod \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\" (UID: \"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a\") " Nov 03 19:08:57.698625 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.698601 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.698714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.698633 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.698714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.698647 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.698714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.698664 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-949r8\" (UniqueName: \"kubernetes.io/projected/f29a321b-415d-413f-9d6e-d38150ef8e67-kube-api-access-949r8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.698714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.698678 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.698714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.698690 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29a321b-415d-413f-9d6e-d38150ef8e67-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.699896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699818 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.699896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699841 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.699896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699856 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.699896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699870 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.699896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699215 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-credential-socket" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.699896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699525 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istiod-ca-cert" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699882 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699933 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.699984 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700004 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700040 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700055 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700069 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700081 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700096 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29swq\" (UniqueName: \"kubernetes.io/projected/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-kube-api-access-29swq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700130 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29a321b-415d-413f-9d6e-d38150ef8e67-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.700144 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15b0fc-c46e-438c-a583-0e69d0dacb5d-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.702965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.702217 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-token" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.722941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.722906 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-kube-api-access-rrclq" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "kube-api-access-rrclq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.723184 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.723041 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-envoy" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.725316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.725272 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-podinfo" pod "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" (UID: "f0eeab63-2d7c-4cb5-9b2c-1135c47d924a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.802788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.801976 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:08:57.803337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.803299 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.803337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.803336 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrclq\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-kube-api-access-rrclq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.803513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.803352 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.803513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.803366 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.803513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.803379 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.803513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.803392 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.808873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.808787 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:08:57.810109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.810042 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:08:57.821040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.820971 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:08:57.845778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.834458 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 19:08:57.881454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.880565 4948 generic.go:356] "Generic (PLEG): container finished" podID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerID="6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8" exitCode=0 Nov 03 19:08:57.881454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.880782 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" event={"ID":"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a","Type":"ContainerDied","Data":"6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8"} Nov 03 19:08:57.881454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.880840 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" event={"ID":"f0eeab63-2d7c-4cb5-9b2c-1135c47d924a","Type":"ContainerDied","Data":"faaa3381cb33ddbcdf0d86767b23f2e389db72d38cdb9fcf4f1bb24329bc956e"} Nov 03 19:08:57.881454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.880872 4948 scope.go:122] "RemoveContainer" containerID="6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8" Nov 03 19:08:57.881454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.881051 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5" Nov 03 19:08:57.887282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.886520 4948 generic.go:356] "Generic (PLEG): container finished" podID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerID="b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd" exitCode=0 Nov 03 19:08:57.887282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.886633 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" event={"ID":"ee56de3f-b47d-4824-9560-96dc8e957dff","Type":"ContainerDied","Data":"b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd"} Nov 03 19:08:57.887282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.886677 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" event={"ID":"ee56de3f-b47d-4824-9560-96dc8e957dff","Type":"ContainerDied","Data":"12a71c065065aa057427997ff907c6e38a47e1ccd4c317a9749cf1d7552afa0e"} Nov 03 19:08:57.887282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.886778 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw" Nov 03 19:08:57.889536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.889191 4948 generic.go:356] "Generic (PLEG): container finished" podID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerID="70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b" exitCode=0 Nov 03 19:08:57.889536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.889305 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" event={"ID":"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc","Type":"ContainerDied","Data":"70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b"} Nov 03 19:08:57.889536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.889345 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" event={"ID":"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc","Type":"ContainerDied","Data":"2299a7aa5a55c3cf96f3b9bd46d065bb9777ed3163b64a17c759305503060758"} Nov 03 19:08:57.889536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.889412 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz" Nov 03 19:08:57.891736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.891470 4948 generic.go:356] "Generic (PLEG): container finished" podID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerID="f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144" exitCode=0 Nov 03 19:08:57.891736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.891544 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" event={"ID":"6f50c444-5018-444d-b775-d40e3d4bf87a","Type":"ContainerDied","Data":"f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144"} Nov 03 19:08:57.891736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.891570 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" event={"ID":"6f50c444-5018-444d-b775-d40e3d4bf87a","Type":"ContainerDied","Data":"afcff45ed272d601807193f492463aad066b9b3792beb919f2af9359c18546db"} Nov 03 19:08:57.891736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.891613 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx" Nov 03 19:08:57.894470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.893957 4948 generic.go:356] "Generic (PLEG): container finished" podID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerID="a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda" exitCode=0 Nov 03 19:08:57.894470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.894055 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" event={"ID":"cd15b0fc-c46e-438c-a583-0e69d0dacb5d","Type":"ContainerDied","Data":"a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda"} Nov 03 19:08:57.894470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.894095 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" event={"ID":"cd15b0fc-c46e-438c-a583-0e69d0dacb5d","Type":"ContainerDied","Data":"32d7f1d4407272545a82871d442e64e1037e5e393cf17d805bd5b726c8eacc34"} Nov 03 19:08:57.894470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.894170 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz" Nov 03 19:08:57.896010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.895985 4948 generic.go:356] "Generic (PLEG): container finished" podID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerID="31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510" exitCode=0 Nov 03 19:08:57.896089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.896074 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" event={"ID":"6637992b-c0a2-4488-b76a-a03ac96768d7","Type":"ContainerDied","Data":"31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510"} Nov 03 19:08:57.896144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.896110 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" event={"ID":"6637992b-c0a2-4488-b76a-a03ac96768d7","Type":"ContainerDied","Data":"f8480b8650352fce76e1ad49977a21e9a59fa6b3d479cf3d3d548995dd55e5b5"} Nov 03 19:08:57.896205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.896187 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv" Nov 03 19:08:57.900962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.900928 4948 generic.go:356] "Generic (PLEG): container finished" podID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerID="0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d" exitCode=0 Nov 03 19:08:57.901066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.901048 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" event={"ID":"f29a321b-415d-413f-9d6e-d38150ef8e67","Type":"ContainerDied","Data":"0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d"} Nov 03 19:08:57.901111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.901091 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" event={"ID":"f29a321b-415d-413f-9d6e-d38150ef8e67","Type":"ContainerDied","Data":"a10b78f4077a908b239cd08ab745aad784351434dbf97aca5d63a31ad4f85c98"} Nov 03 19:08:57.901169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.901157 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt" Nov 03 19:08:57.903407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903382 4948 scope.go:122] "RemoveContainer" containerID="8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81" Nov 03 19:08:57.903804 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903780 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-data\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.903927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903816 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-certs\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.903927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903851 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-envoy\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.903927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903885 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6637992b-c0a2-4488-b76a-a03ac96768d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6637992b-c0a2-4488-b76a-a03ac96768d7-istiod-ca-cert\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.903927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903905 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-podinfo\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903931 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istiod-ca-cert\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903954 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-token\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-token\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.903991 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-token\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-token\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904024 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-socket\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904038 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-data" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904064 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-kube-api-access-z749f\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-kube-api-access-z749f\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904091 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-token\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-token\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904122 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istiod-ca-cert\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904131 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-certs" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.904156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904146 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-podinfo\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904226 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-certs\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904309 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-podinfo\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904347 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f50c444-5018-444d-b775-d40e3d4bf87a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f50c444-5018-444d-b775-d40e3d4bf87a-istiod-ca-cert\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904388 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee56de3f-b47d-4824-9560-96dc8e957dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56de3f-b47d-4824-9560-96dc8e957dff-istiod-ca-cert\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904423 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-credential-socket\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904454 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-socket\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904516 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-token\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-token\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.904678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.904607 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6637992b-c0a2-4488-b76a-a03ac96768d7-istiod-ca-cert" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.905132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.905014 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-data\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.905384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.905365 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-socket\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.907287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907263 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-certs\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.907409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907315 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-kube-api-access-z6r4t\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-kube-api-access-z6r4t\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.907409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907352 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-credential-socket\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.907589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907429 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" Nov 03 19:08:57.907589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907440 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-podinfo\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.907589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907488 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-socket\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.907589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907528 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-kube-api-access-mgxm7\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-kube-api-access-mgxm7\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.907589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907573 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-certs\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.907968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907615 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-certs\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.907968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907677 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-data\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.907968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907783 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-kube-api-access-j2dhg\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-kube-api-access-j2dhg\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.907968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907834 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-envoy\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.907968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907874 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-credential-socket\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.907968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907924 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-credential-socket\") pod \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\" (UID: \"e98895b9-c0ff-4d5a-b615-ba0a0227f1fc\") " Nov 03 19:08:57.907968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907958 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-data\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908006 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-podinfo\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908084 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-data\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908117 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-token\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-token\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908200 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-socket\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908245 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-credential-socket\") pod \"6637992b-c0a2-4488-b76a-a03ac96768d7\" (UID: \"6637992b-c0a2-4488-b76a-a03ac96768d7\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908284 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-envoy\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908317 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-kube-api-access-j9rjv\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-kube-api-access-j9rjv\") pod \"6f50c444-5018-444d-b775-d40e3d4bf87a\" (UID: \"6f50c444-5018-444d-b775-d40e3d4bf87a\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908405 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-envoy\") pod \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\" (UID: \"4f639da3-66ce-4c7b-b94a-5f9edbf6651c\") " Nov 03 19:08:57.908457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908439 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-envoy\") pod \"ee56de3f-b47d-4824-9560-96dc8e957dff\" (UID: \"ee56de3f-b47d-4824-9560-96dc8e957dff\") " Nov 03 19:08:57.909096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.906060 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istiod-ca-cert" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.909096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907773 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-socket" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.909096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907992 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-certs" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.909096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908280 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-socket" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.909096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.908450 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-certs" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.909096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907143 4948 generic.go:356] "Generic (PLEG): container finished" podID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerID="7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630" exitCode=0 Nov 03 19:08:57.909096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909035 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istiod-ca-cert" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909268 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-data" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909451 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909473 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909489 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6637992b-c0a2-4488-b76a-a03ac96768d7-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909506 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909521 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909533 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909547 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909559 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909571 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.909662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909584 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:57.910736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.907161 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" event={"ID":"4f639da3-66ce-4c7b-b94a-5f9edbf6651c","Type":"ContainerDied","Data":"7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630"} Nov 03 19:08:57.910736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.909938 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5" event={"ID":"4f639da3-66ce-4c7b-b94a-5f9edbf6651c","Type":"ContainerDied","Data":"52e647fd85bd7bf0ce9c920b20c58da4aa46f5301321b4b7c36e335d4a4d8c46"} Nov 03 19:08:57.910736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.910126 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-credential-socket" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.910736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.910354 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee56de3f-b47d-4824-9560-96dc8e957dff-istiod-ca-cert" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.910736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.910434 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-credential-socket" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.910736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.910630 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-socket" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.910736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.910544 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f50c444-5018-444d-b775-d40e3d4bf87a-istiod-ca-cert" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:57.911252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.910866 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-credential-socket" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.912252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.911749 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-data" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.912721 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-data" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.914084 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-socket" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.914124 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-credential-socket" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.914176 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-certs" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.914367 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-kube-api-access-z6r4t" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "kube-api-access-z6r4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.914599 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-credential-socket" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.915348 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-certs" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.915582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.915377 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-socket" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.916340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.915737 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-data" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.916340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.916107 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-envoy" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.918204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.918107 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-token" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.919027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.919005 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-token" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.919125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.919103 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-token" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.919196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.919181 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-podinfo" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.919281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.919264 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-kube-api-access-j2dhg" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "kube-api-access-j2dhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.921331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.921308 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-envoy" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.924900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.924882 4948 scope.go:122] "RemoveContainer" containerID="6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8" Nov 03 19:08:57.925041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.925020 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-podinfo" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.925120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.925109 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-token" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.925315 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:57.925294 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8\": container with ID starting with 6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8 not found: ID does not exist" containerID="6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8" Nov 03 19:08:57.925367 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.925328 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8"} err="failed to get container status \"6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8\": rpc error: code = NotFound desc = could not find container \"6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8\": container with ID starting with 6b5db3c6b98d3a0cc55c40dc93b97383a40813364f791cb453565816138043f8 not found: ID does not exist" Nov 03 19:08:57.925367 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.925350 4948 scope.go:122] "RemoveContainer" containerID="8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81" Nov 03 19:08:57.925614 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:57.925591 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81\": container with ID starting with 8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81 not found: ID does not exist" containerID="8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81" Nov 03 19:08:57.925716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.925623 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81"} err="failed to get container status \"8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81\": rpc error: code = NotFound desc = could not find container \"8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81\": container with ID starting with 8c83fd94723a57e199d431223ad4847ea53eb2acd1bb566742675fac7e1a5f81 not found: ID does not exist" Nov 03 19:08:57.925716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.925643 4948 scope.go:122] "RemoveContainer" containerID="b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd" Nov 03 19:08:57.925851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.925829 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-kube-api-access-z749f" pod "6637992b-c0a2-4488-b76a-a03ac96768d7" (UID: "6637992b-c0a2-4488-b76a-a03ac96768d7"). InnerVolumeSpecName "kube-api-access-z749f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.938635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938355 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-envoy" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.938635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938396 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-podinfo" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.938635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938402 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-kube-api-access-j9rjv" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "kube-api-access-j9rjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.938635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938407 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-envoy" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.938927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938664 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-podinfo" pod "6f50c444-5018-444d-b775-d40e3d4bf87a" (UID: "6f50c444-5018-444d-b775-d40e3d4bf87a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.938927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938713 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-envoy" pod "4f639da3-66ce-4c7b-b94a-5f9edbf6651c" (UID: "4f639da3-66ce-4c7b-b94a-5f9edbf6651c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:57.938927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938918 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-token" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.939028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938932 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-podinfo" pod "ee56de3f-b47d-4824-9560-96dc8e957dff" (UID: "ee56de3f-b47d-4824-9560-96dc8e957dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:57.939028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.938943 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-kube-api-access-mgxm7" pod "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" (UID: "e98895b9-c0ff-4d5a-b615-ba0a0227f1fc"). InnerVolumeSpecName "kube-api-access-mgxm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:57.941227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.941210 4948 scope.go:122] "RemoveContainer" containerID="b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd" Nov 03 19:08:57.941566 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:57.941542 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd\": container with ID starting with b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd not found: ID does not exist" containerID="b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd" Nov 03 19:08:57.941631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.941574 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd"} err="failed to get container status \"b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd\": rpc error: code = NotFound desc = could not find container \"b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd\": container with ID starting with b528cd28f3d49c549f1d9bd9a80839bd86d6016980ce674e3ba82344122deecd not found: ID does not exist" Nov 03 19:08:57.941631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.941594 4948 scope.go:122] "RemoveContainer" containerID="70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b" Nov 03 19:08:57.965657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.965618 4948 scope.go:122] "RemoveContainer" containerID="70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b" Nov 03 19:08:57.972888 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:57.972852 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b\": container with ID starting with 70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b not found: ID does not exist" containerID="70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b" Nov 03 19:08:57.973040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.972902 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b"} err="failed to get container status \"70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b\": rpc error: code = NotFound desc = could not find container \"70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b\": container with ID starting with 70c8c27d3d58219bc2b36f321eb2a693ef204deafab18c3b82ac20256f9b9f4b not found: ID does not exist" Nov 03 19:08:57.973040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.972936 4948 scope.go:122] "RemoveContainer" containerID="f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144" Nov 03 19:08:57.993874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.993671 4948 scope.go:122] "RemoveContainer" containerID="f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144" Nov 03 19:08:57.994362 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:57.994335 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144\": container with ID starting with f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144 not found: ID does not exist" containerID="f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144" Nov 03 19:08:57.994498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.994376 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144"} err="failed to get container status \"f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144\": rpc error: code = NotFound desc = could not find container \"f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144\": container with ID starting with f4d3e905e19f4399e4cf3cb2b1dd4e6a5c0656b3e0bad3bd27f06aaab3bf4144 not found: ID does not exist" Nov 03 19:08:57.994498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:57.994407 4948 scope.go:122] "RemoveContainer" containerID="a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda" Nov 03 19:08:58.008054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.008029 4948 scope.go:122] "RemoveContainer" containerID="a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda" Nov 03 19:08:58.008383 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:58.008354 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda\": container with ID starting with a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda not found: ID does not exist" containerID="a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda" Nov 03 19:08:58.008476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.008391 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda"} err="failed to get container status \"a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda\": rpc error: code = NotFound desc = could not find container \"a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda\": container with ID starting with a6b1ead2c16ca762cdfe63ff323887db46262a6a584729a42988dd7e5defaeda not found: ID does not exist" Nov 03 19:08:58.008476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.008418 4948 scope.go:122] "RemoveContainer" containerID="31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510" Nov 03 19:08:58.011740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011692 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.011842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011746 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.011842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011763 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z749f\" (UniqueName: \"kubernetes.io/projected/6637992b-c0a2-4488-b76a-a03ac96768d7-kube-api-access-z749f\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.011842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011778 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.011842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011792 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.011842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011806 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.011842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011820 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f50c444-5018-444d-b775-d40e3d4bf87a-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.011842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011835 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56de3f-b47d-4824-9560-96dc8e957dff-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011848 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011862 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011876 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011892 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011906 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011923 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6r4t\" (UniqueName: \"kubernetes.io/projected/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-kube-api-access-z6r4t\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011938 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011952 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011966 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011981 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxm7\" (UniqueName: \"kubernetes.io/projected/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-kube-api-access-mgxm7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.011995 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012009 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dhg\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-kube-api-access-j2dhg\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012023 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012036 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012051 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012067 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012081 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012095 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012109 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012123 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012137 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012150 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f50c444-5018-444d-b775-d40e3d4bf87a-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012163 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rjv\" (UniqueName: \"kubernetes.io/projected/6f50c444-5018-444d-b775-d40e3d4bf87a-kube-api-access-j9rjv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012176 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012191 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56de3f-b47d-4824-9560-96dc8e957dff-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012205 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6637992b-c0a2-4488-b76a-a03ac96768d7-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.012660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.012219 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f639da3-66ce-4c7b-b94a-5f9edbf6651c-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:58.021168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.021146 4948 scope.go:122] "RemoveContainer" containerID="31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510" Nov 03 19:08:58.021505 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:58.021481 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510\": container with ID starting with 31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510 not found: ID does not exist" containerID="31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510" Nov 03 19:08:58.021576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.021515 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510"} err="failed to get container status \"31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510\": rpc error: code = NotFound desc = could not find container \"31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510\": container with ID starting with 31e4823c36f0c5a0c55a0a88573316ffc85916ec9c49d04e50dc1acef9dd2510 not found: ID does not exist" Nov 03 19:08:58.021576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.021539 4948 scope.go:122] "RemoveContainer" containerID="0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d" Nov 03 19:08:58.033206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.033184 4948 scope.go:122] "RemoveContainer" containerID="0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d" Nov 03 19:08:58.033523 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:58.033493 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d\": container with ID starting with 0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d not found: ID does not exist" containerID="0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d" Nov 03 19:08:58.033597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.033529 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d"} err="failed to get container status \"0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d\": rpc error: code = NotFound desc = could not find container \"0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d\": container with ID starting with 0e305168e4678c81aedfa9c6d1b054908e662db7c5cf8a2717a7c44391ed701d not found: ID does not exist" Nov 03 19:08:58.033597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.033543 4948 scope.go:122] "RemoveContainer" containerID="7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630" Nov 03 19:08:58.047211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.047191 4948 scope.go:122] "RemoveContainer" containerID="70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90" Nov 03 19:08:58.136732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.136685 4948 scope.go:122] "RemoveContainer" containerID="7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630" Nov 03 19:08:58.137060 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:58.137037 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630\": container with ID starting with 7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630 not found: ID does not exist" containerID="7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630" Nov 03 19:08:58.137155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.137067 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630"} err="failed to get container status \"7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630\": rpc error: code = NotFound desc = could not find container \"7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630\": container with ID starting with 7b3d22c4b33c4d4d6af48b5b3c147371818d0bb362eb0c36d50847689b16f630 not found: ID does not exist" Nov 03 19:08:58.137155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.137085 4948 scope.go:122] "RemoveContainer" containerID="70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90" Nov 03 19:08:58.137387 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:08:58.137368 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90\": container with ID starting with 70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90 not found: ID does not exist" containerID="70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90" Nov 03 19:08:58.137459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.137391 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90"} err="failed to get container status \"70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90\": rpc error: code = NotFound desc = could not find container \"70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90\": container with ID starting with 70c851a5bb6e59ed87fea89cadee84fea2de07c9ba07f83a752cc210532fdf90 not found: ID does not exist" Nov 03 19:08:58.343388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.343342 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz"] Nov 03 19:08:58.348096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.348066 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-ql4lz"] Nov 03 19:08:58.760612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.760577 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" path="/var/lib/kubelet/pods/cd15b0fc-c46e-438c-a583-0e69d0dacb5d/volumes" Nov 03 19:08:58.825041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.825007 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt"] Nov 03 19:08:58.832264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.832232 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-qkdmt"] Nov 03 19:08:58.940486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.940451 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5"] Nov 03 19:08:58.945128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:58.945102 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-cxpf5"] Nov 03 19:08:59.058333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.058247 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx"] Nov 03 19:08:59.062912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.062884 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-n5xtx"] Nov 03 19:08:59.176255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.176225 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz"] Nov 03 19:08:59.177141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.177118 4948 scope.go:122] "RemoveContainer" containerID="367083fab54c6da23838d2bc3c2fd88cb530e3d2b70f6163837b89eec2f372aa" Nov 03 19:08:59.187654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.187615 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-592dz"] Nov 03 19:08:59.204504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.204476 4948 scope.go:122] "RemoveContainer" containerID="c95913aea9c9bb82eea3704c00bb0c4c98547f92c52603f898078a6145d58cf4" Nov 03 19:08:59.215544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.215523 4948 scope.go:122] "RemoveContainer" containerID="6c4c3cd349aaef577a77d19f8ae9f7a379f9d7c15b50a2e6ab695ab6e811cd1e" Nov 03 19:08:59.225560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.225537 4948 scope.go:122] "RemoveContainer" containerID="f50e2c746fdb6de8643ee02a0814dc0ff59dd5f3d5f3ffd1fde9ab83cb90d516" Nov 03 19:08:59.236020 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.235999 4948 scope.go:122] "RemoveContainer" containerID="33b9d66dc15e17d46250eced26b63ce7f20feb87d9380c5795c405a411a5e23f" Nov 03 19:08:59.245545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.245525 4948 scope.go:122] "RemoveContainer" containerID="eb0e974cde947ac33ed566cb3c91771e65db4b1413622d7b99df7aec1618fc12" Nov 03 19:08:59.256758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.256690 4948 scope.go:122] "RemoveContainer" containerID="83d7b9f3ea32a2914d848f142e51259915dd3434ffdd0d6f25637fe8bdc2912e" Nov 03 19:08:59.276320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.276294 4948 scope.go:122] "RemoveContainer" containerID="09ba816d6f9efa711d9a3b23cc2933dbc13e1491b1612b0d5e155403aeeb99bb" Nov 03 19:08:59.289199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.289173 4948 scope.go:122] "RemoveContainer" containerID="56445504831ea0d5a09cce9b6307c04a251f6d0b93146c9e15c1ea3096fe971d" Nov 03 19:08:59.293021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.292994 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv"] Nov 03 19:08:59.298616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.298593 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-szxrv"] Nov 03 19:08:59.411290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.411254 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5"] Nov 03 19:08:59.416687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.416658 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-vdgg5"] Nov 03 19:08:59.527031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.526997 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw"] Nov 03 19:08:59.532745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:08:59.532716 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-tbwhw"] Nov 03 19:09:00.761297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:00.761261 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" path="/var/lib/kubelet/pods/4f639da3-66ce-4c7b-b94a-5f9edbf6651c/volumes" Nov 03 19:09:00.761977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:00.761957 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" path="/var/lib/kubelet/pods/6637992b-c0a2-4488-b76a-a03ac96768d7/volumes" Nov 03 19:09:00.762605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:00.762587 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" path="/var/lib/kubelet/pods/6f50c444-5018-444d-b775-d40e3d4bf87a/volumes" Nov 03 19:09:00.763640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:00.763621 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" path="/var/lib/kubelet/pods/e98895b9-c0ff-4d5a-b615-ba0a0227f1fc/volumes" Nov 03 19:09:00.764281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:00.764263 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" path="/var/lib/kubelet/pods/ee56de3f-b47d-4824-9560-96dc8e957dff/volumes" Nov 03 19:09:00.764920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:00.764903 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" path="/var/lib/kubelet/pods/f0eeab63-2d7c-4cb5-9b2c-1135c47d924a/volumes" Nov 03 19:09:00.765936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:00.765917 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" path="/var/lib/kubelet/pods/f29a321b-415d-413f-9d6e-d38150ef8e67/volumes" Nov 03 19:09:02.799995 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:02.799955 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:09:04.543788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.543754 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch"] Nov 03 19:09:04.544533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544515 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" Nov 03 19:09:04.544533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544533 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544550 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544556 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544565 4948 cpu_manager.go:414] "containerMap: removing container" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544572 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544586 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544592 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544604 4948 cpu_manager.go:414] "containerMap: removing container" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544608 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544617 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544622 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544632 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544636 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544646 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" Nov 03 19:09:04.544650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544650 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544664 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544668 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544693 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544709 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544717 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544721 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544733 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544737 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544742 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544746 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544852 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd15b0fc-c46e-438c-a583-0e69d0dacb5d" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544861 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee56de3f-b47d-4824-9560-96dc8e957dff" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544867 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b567e2b-d032-4733-a6bb-6531f0d149a2" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544875 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1e7095-d0f7-4548-ae01-1ea485ece334" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544880 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544886 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98895b9-c0ff-4d5a-b615-ba0a0227f1fc" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544895 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544900 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29a321b-415d-413f-9d6e-d38150ef8e67" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544906 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="6637992b-c0a2-4488-b76a-a03ac96768d7" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544912 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4b1269-cfed-484b-8957-d8eb9b3dbb43" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544919 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f50c444-5018-444d-b775-d40e3d4bf87a" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544927 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" Nov 03 19:09:04.545086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.544933 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="651bbe89-2f02-462c-bc24-cf5a69fbd101" containerName="istio-proxy" Nov 03 19:09:04.550215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.550196 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.604473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.604442 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch"] Nov 03 19:09:04.626904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.626872 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.627075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.626946 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.627161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.627135 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.627292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.627275 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mw5x\" (UniqueName: \"kubernetes.io/projected/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-kube-api-access-9mw5x\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.650047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.650019 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ecf40a-serving-cert\"" Nov 03 19:09:04.650239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.650067 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:09:04.650239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.650111 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\"" Nov 03 19:09:04.654864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.654839 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:09:04.654999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.654947 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 19:09:04.729095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.729063 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.729256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.729121 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.729256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.729173 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.729256 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:04.729213 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-serving-cert: secret "maas-api-e2e-shared-a-w0-ecf40a-serving-cert" not found Nov 03 19:09:04.729373 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.729220 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mw5x\" (UniqueName: \"kubernetes.io/projected/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-kube-api-access-9mw5x\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.729373 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:04.729291 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls podName:e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698 nodeName:}" failed. No retries permitted until 2026-11-03 19:09:05.229270345 +0000 UTC m=+1874.712266252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls") pod "maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" (UID: "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698") : secret "maas-api-e2e-shared-a-w0-ecf40a-serving-cert" not found Nov 03 19:09:04.729904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.729878 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.731494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.731478 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:04.800940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:04.800266 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mw5x\" (UniqueName: \"kubernetes.io/projected/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-kube-api-access-9mw5x\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:05.239466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:05.239423 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:05.241845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:05.241814 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:05.466534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:05.466497 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:05.857999 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:09:05.857967 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a3feeb_2e84_4e4e_b944_4fcdd5c5f698.slice/crio-217ecddceeacd97a90ac037855f2a8c277f6fbcaa4ce3f980db4b901e22bc479 WatchSource:0}: Error finding container 217ecddceeacd97a90ac037855f2a8c277f6fbcaa4ce3f980db4b901e22bc479: Status 404 returned error can't find the container with id 217ecddceeacd97a90ac037855f2a8c277f6fbcaa4ce3f980db4b901e22bc479 Nov 03 19:09:05.858742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:05.858654 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch"] Nov 03 19:09:05.860352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:05.860309 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:09:06.019640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:06.019606 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" event={"ID":"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698","Type":"ContainerStarted","Data":"217ecddceeacd97a90ac037855f2a8c277f6fbcaa4ce3f980db4b901e22bc479"} Nov 03 19:09:09.079254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.079211 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" event={"ID":"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698","Type":"ContainerStarted","Data":"7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599"} Nov 03 19:09:09.080358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.080335 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:09.215726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.210772 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" podStartSLOduration=2.8441808589999997 podStartE2EDuration="5.210750854s" totalImagesPullingTime="2.366569995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:09.208195199 +0000 UTC m=+1878.691191117" watchObservedRunningTime="2026-11-03 19:09:09.210750854 +0000 UTC m=+1878.693746769" Nov 03 19:09:09.285723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.284831 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58"] Nov 03 19:09:09.294836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.286025 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" Nov 03 19:09:09.294836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.286053 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0eeab63-2d7c-4cb5-9b2c-1135c47d924a" containerName="istio-proxy" Nov 03 19:09:09.294836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.286301 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f639da3-66ce-4c7b-b94a-5f9edbf6651c" containerName="istio-proxy" Nov 03 19:09:09.294836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.293995 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.418727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.409662 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e1f8f8-openshift-default-dockercfg-vmfrf\"" Nov 03 19:09:09.418727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.411205 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58"] Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.502844 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-podinfo\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.502899 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d57cd9d-8092-445f-8a1a-b8773a31479f-istiod-ca-cert\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.503086 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-data\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.503163 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-token\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.503190 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-892lm\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-kube-api-access-892lm\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.503308 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-certs\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.503342 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-socket\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.503364 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-envoy\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.522720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.504684 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-credential-socket\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.605865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.605777 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-podinfo\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.605865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.605820 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d57cd9d-8092-445f-8a1a-b8773a31479f-istiod-ca-cert\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.605911 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-data\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.605949 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-token\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.605965 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-892lm\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-kube-api-access-892lm\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.606045 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-certs\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.606073 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-socket\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.606097 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-envoy\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.606171 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-credential-socket\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.606798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.606770 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d57cd9d-8092-445f-8a1a-b8773a31479f-istiod-ca-cert\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.607018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.606993 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-socket\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.607087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.607043 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-credential-socket\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.607087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.607055 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-certs\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.607174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.607152 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-data\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.608547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.608526 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-podinfo\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.609264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.609244 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-envoy\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.667822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.667792 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-token\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.668013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.667971 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-892lm\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-kube-api-access-892lm\") pod \"e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:09.744465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:09.744428 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:10.183259 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:10.183211 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58"] Nov 03 19:09:10.219111 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:09:10.219083 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d57cd9d_8092_445f_8a1a_b8773a31479f.slice/crio-a70ef35c6614da83fa93a895e37969609f30b786e338b73e07d423e085a34a3b WatchSource:0}: Error finding container a70ef35c6614da83fa93a895e37969609f30b786e338b73e07d423e085a34a3b: Status 404 returned error can't find the container with id a70ef35c6614da83fa93a895e37969609f30b786e338b73e07d423e085a34a3b Nov 03 19:09:10.221669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:10.221637 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:10.221771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:10.221724 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:10.221771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:10.221753 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:09:11.100054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:11.100009 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" event={"ID":"3d57cd9d-8092-445f-8a1a-b8773a31479f","Type":"ContainerStarted","Data":"be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1"} Nov 03 19:09:11.100266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:11.100066 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" event={"ID":"3d57cd9d-8092-445f-8a1a-b8773a31479f","Type":"ContainerStarted","Data":"a70ef35c6614da83fa93a895e37969609f30b786e338b73e07d423e085a34a3b"} Nov 03 19:09:11.228576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:11.228516 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podStartSLOduration=2.22850032 podStartE2EDuration="2.22850032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:11.222508648 +0000 UTC m=+1880.705504562" watchObservedRunningTime="2026-11-03 19:09:11.22850032 +0000 UTC m=+1880.711496288" Nov 03 19:09:11.745448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:11.745408 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:11.752046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:11.752004 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:12.111073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:12.110979 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:12.112112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:12.112088 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:09:16.106959 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:16.106930 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:09:17.666821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:17.666674 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs"] Nov 03 19:09:17.672313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:17.672286 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:17.761757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:17.761384 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e1f8f8\"" Nov 03 19:09:17.779749 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:17.764142 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e1f8f8-dockercfg-nmzmb\"" Nov 03 19:09:17.889733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:17.807578 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvsn\" (UniqueName: \"kubernetes.io/projected/b165948d-4b0c-4df0-b225-3675562ae7e3-kube-api-access-lvvsn\") pod \"payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:17.901728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:17.890216 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b165948d-4b0c-4df0-b225-3675562ae7e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:17.947775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:17.864925 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs"] Nov 03 19:09:18.022437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.020054 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b165948d-4b0c-4df0-b225-3675562ae7e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:18.023091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.023058 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvsn\" (UniqueName: \"kubernetes.io/projected/b165948d-4b0c-4df0-b225-3675562ae7e3-kube-api-access-lvvsn\") pod \"payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:18.024669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.021066 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b165948d-4b0c-4df0-b225-3675562ae7e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:18.025353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.025333 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62"] Nov 03 19:09:18.035441 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.035400 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62"] Nov 03 19:09:18.036037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.035930 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.086576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.086546 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvsn\" (UniqueName: \"kubernetes.io/projected/b165948d-4b0c-4df0-b225-3675562ae7e3-kube-api-access-lvvsn\") pod \"payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:18.125249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.125211 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be04bddc-7158-41f5-9556-cef105850742-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.125462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.125440 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkzc\" (UniqueName: \"kubernetes.io/projected/be04bddc-7158-41f5-9556-cef105850742-kube-api-access-ljkzc\") pod \"payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.227044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.226948 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkzc\" (UniqueName: \"kubernetes.io/projected/be04bddc-7158-41f5-9556-cef105850742-kube-api-access-ljkzc\") pod \"payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.227217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.227082 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be04bddc-7158-41f5-9556-cef105850742-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.227910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.227889 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be04bddc-7158-41f5-9556-cef105850742-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.289437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.289398 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkzc\" (UniqueName: \"kubernetes.io/projected/be04bddc-7158-41f5-9556-cef105850742-kube-api-access-ljkzc\") pod \"payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.297150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.297120 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:18.358347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.358320 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:18.683069 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:09:18.683037 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb165948d_4b0c_4df0_b225_3675562ae7e3.slice/crio-2adc0fc023263b5e2abd861506efeb1d64b071b6cefc7658207faed5218227d2 WatchSource:0}: Error finding container 2adc0fc023263b5e2abd861506efeb1d64b071b6cefc7658207faed5218227d2: Status 404 returned error can't find the container with id 2adc0fc023263b5e2abd861506efeb1d64b071b6cefc7658207faed5218227d2 Nov 03 19:09:18.683404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.683383 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs"] Nov 03 19:09:18.764026 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:09:18.763994 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe04bddc_7158_41f5_9556_cef105850742.slice/crio-fb87cbfae13539baba33edaf63961332ffc0e09946f6d3f8b4aae43727dce44a WatchSource:0}: Error finding container fb87cbfae13539baba33edaf63961332ffc0e09946f6d3f8b4aae43727dce44a: Status 404 returned error can't find the container with id fb87cbfae13539baba33edaf63961332ffc0e09946f6d3f8b4aae43727dce44a Nov 03 19:09:18.764609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:18.764587 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62"] Nov 03 19:09:19.177365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.177322 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" event={"ID":"b165948d-4b0c-4df0-b225-3675562ae7e3","Type":"ContainerStarted","Data":"f3f3086442e2725d60e024b0a7f1c83aeddcc3ffe71a505b5057a54ab0575659"} Nov 03 19:09:19.177577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.177388 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:19.177577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.177415 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" event={"ID":"b165948d-4b0c-4df0-b225-3675562ae7e3","Type":"ContainerStarted","Data":"2adc0fc023263b5e2abd861506efeb1d64b071b6cefc7658207faed5218227d2"} Nov 03 19:09:19.179339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.179313 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" event={"ID":"be04bddc-7158-41f5-9556-cef105850742","Type":"ContainerStarted","Data":"dafec5197639b65495ca8012285aef277fbe1d84a5e4c71a2750aac8fc9c66ed"} Nov 03 19:09:19.179462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.179357 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" event={"ID":"be04bddc-7158-41f5-9556-cef105850742","Type":"ContainerStarted","Data":"fb87cbfae13539baba33edaf63961332ffc0e09946f6d3f8b4aae43727dce44a"} Nov 03 19:09:19.179462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.179409 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:19.295483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.295418 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" podStartSLOduration=2.2953989249999998 podStartE2EDuration="2.295398925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:19.294111958 +0000 UTC m=+1888.777107873" watchObservedRunningTime="2026-11-03 19:09:19.295398925 +0000 UTC m=+1888.778394841" Nov 03 19:09:19.434744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:19.434154 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" podStartSLOduration=2.434084504 podStartE2EDuration="2.434084504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:19.423240173 +0000 UTC m=+1888.906236088" watchObservedRunningTime="2026-11-03 19:09:19.434084504 +0000 UTC m=+1888.917080419" Nov 03 19:09:20.179991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.179958 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2"] Nov 03 19:09:20.184728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.184690 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.238669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.238634 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2"] Nov 03 19:09:20.262622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.262585 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.263223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.263194 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-metrics-certs\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.263426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.263407 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4c78c-c217-415f-82cc-683275be6bea-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.263570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.263554 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzss\" (UniqueName: \"kubernetes.io/projected/74a4c78c-c217-415f-82cc-683275be6bea-kube-api-access-gkzss\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.285268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.285235 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ba5674-serving-cert\"" Nov 03 19:09:20.365412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.365359 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzss\" (UniqueName: \"kubernetes.io/projected/74a4c78c-c217-415f-82cc-683275be6bea-kube-api-access-gkzss\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.365583 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.365502 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.365882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.365847 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-metrics-certs\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.365978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.365961 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4c78c-c217-415f-82cc-683275be6bea-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.366151 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:20.366130 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-serving-cert: secret "maas-api-e2e-api-a-ba5674-serving-cert" not found Nov 03 19:09:20.366215 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:20.366203 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls podName:74a4c78c-c217-415f-82cc-683275be6bea nodeName:}" failed. No retries permitted until 2026-11-03 19:09:20.866182683 +0000 UTC m=+1890.349178578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls") pod "maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" (UID: "74a4c78c-c217-415f-82cc-683275be6bea") : secret "maas-api-e2e-api-a-ba5674-serving-cert" not found Nov 03 19:09:20.367238 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.367188 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4c78c-c217-415f-82cc-683275be6bea-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.370273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.370241 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-metrics-certs\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.430192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.430121 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzss\" (UniqueName: \"kubernetes.io/projected/74a4c78c-c217-415f-82cc-683275be6bea-kube-api-access-gkzss\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.875512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.875470 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:20.877954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:20.877929 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls\") pod \"maas-api-e2e-api-a-ba5674-5f8969588-8pxr2\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:21.103818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:21.103781 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:21.487735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:21.487694 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2"] Nov 03 19:09:21.490403 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:09:21.490371 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a4c78c_c217_415f_82cc_683275be6bea.slice/crio-70ca036d025c02d95259d2641f492c8ac4aa63aa0ad3048ca306f0383f18c50e WatchSource:0}: Error finding container 70ca036d025c02d95259d2641f492c8ac4aa63aa0ad3048ca306f0383f18c50e: Status 404 returned error can't find the container with id 70ca036d025c02d95259d2641f492c8ac4aa63aa0ad3048ca306f0383f18c50e Nov 03 19:09:22.209990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:22.209957 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" event={"ID":"74a4c78c-c217-415f-82cc-683275be6bea","Type":"ContainerStarted","Data":"70ca036d025c02d95259d2641f492c8ac4aa63aa0ad3048ca306f0383f18c50e"} Nov 03 19:09:23.222406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:23.222369 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" event={"ID":"74a4c78c-c217-415f-82cc-683275be6bea","Type":"ContainerStarted","Data":"a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4"} Nov 03 19:09:23.222943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:23.222448 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:23.348254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:23.348193 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" podStartSLOduration=2.415039189 podStartE2EDuration="3.348178735s" totalImagesPullingTime="933.139546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:23.344069418 +0000 UTC m=+1892.827065331" watchObservedRunningTime="2026-11-03 19:09:23.348178735 +0000 UTC m=+1892.831174649" Nov 03 19:09:29.249234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:29.249198 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:09:30.196109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:30.196076 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:09:30.196331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:30.196170 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:09:32.801123 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:32.801078 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:09:34.051431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.051395 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4"] Nov 03 19:09:34.101925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.101881 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ac4a0a-662n4"] Nov 03 19:09:34.513361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.513322 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm"] Nov 03 19:09:34.519429 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.519402 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.571654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.571617 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm"] Nov 03 19:09:34.572793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.572762 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b485d8-serving-cert\"" Nov 03 19:09:34.598191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.598159 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9924w\" (UniqueName: \"kubernetes.io/projected/9ecbc954-3a56-490f-9087-9422730083ee-kube-api-access-9924w\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.598360 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.598202 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.598409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.598351 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ecbc954-3a56-490f-9087-9422730083ee-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.598409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.598404 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.700221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.700189 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9924w\" (UniqueName: \"kubernetes.io/projected/9ecbc954-3a56-490f-9087-9422730083ee-kube-api-access-9924w\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.700221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.700232 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.700467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.700416 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ecbc954-3a56-490f-9087-9422730083ee-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.700570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.700551 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.700691 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:34.700673 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-serving-cert: secret "maas-api-e2e-shared-b-w2-b485d8-serving-cert" not found Nov 03 19:09:34.700761 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:09:34.700749 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls podName:9ecbc954-3a56-490f-9087-9422730083ee nodeName:}" failed. No retries permitted until 2026-11-03 19:09:35.200734181 +0000 UTC m=+1904.683730073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls") pod "maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" (UID: "9ecbc954-3a56-490f-9087-9422730083ee") : secret "maas-api-e2e-shared-b-w2-b485d8-serving-cert" not found Nov 03 19:09:34.701150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.701128 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ecbc954-3a56-490f-9087-9422730083ee-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.702983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.702963 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.820792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.818512 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9924w\" (UniqueName: \"kubernetes.io/projected/9ecbc954-3a56-490f-9087-9422730083ee-kube-api-access-9924w\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:34.822978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:34.822941 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049a88a5-7b87-4324-9b8c-ea46d263756f" path="/var/lib/kubelet/pods/049a88a5-7b87-4324-9b8c-ea46d263756f/volumes" Nov 03 19:09:35.216868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:35.216834 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:35.219501 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:35.219472 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:35.435338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:35.435309 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:35.825337 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:09:35.825304 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecbc954_3a56_490f_9087_9422730083ee.slice/crio-cbdd97ae2e204d7ae5d8ccc12b83c492a32d5e821f634423b013d7d4fa10ec13 WatchSource:0}: Error finding container cbdd97ae2e204d7ae5d8ccc12b83c492a32d5e821f634423b013d7d4fa10ec13: Status 404 returned error can't find the container with id cbdd97ae2e204d7ae5d8ccc12b83c492a32d5e821f634423b013d7d4fa10ec13 Nov 03 19:09:35.825854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:35.825827 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm"] Nov 03 19:09:36.446649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:36.446607 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" event={"ID":"9ecbc954-3a56-490f-9087-9422730083ee","Type":"ContainerStarted","Data":"cbdd97ae2e204d7ae5d8ccc12b83c492a32d5e821f634423b013d7d4fa10ec13"} Nov 03 19:09:37.504905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:37.504865 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" event={"ID":"9ecbc954-3a56-490f-9087-9422730083ee","Type":"ContainerStarted","Data":"8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f"} Nov 03 19:09:37.507124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:37.507093 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:37.629795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:37.629739 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" podStartSLOduration=2.835391308 podStartE2EDuration="3.629717555s" totalImagesPullingTime="794.326247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:37.627310171 +0000 UTC m=+1907.110306086" watchObservedRunningTime="2026-11-03 19:09:37.629717555 +0000 UTC m=+1907.112713470" Nov 03 19:09:43.988306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:43.988266 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf"] Nov 03 19:09:43.993442 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:43.993421 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.100409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.100366 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf"] Nov 03 19:09:44.101240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.101213 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:09:44.102159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.102087 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:09:44.103466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.102111 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.103766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.103737 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r958\" (UniqueName: \"kubernetes.io/projected/937a0f9e-1110-490c-9034-3b5c67f8696a-kube-api-access-5r958\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.103889 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.103852 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.205100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.205065 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.205296 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.205195 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r958\" (UniqueName: \"kubernetes.io/projected/937a0f9e-1110-490c-9034-3b5c67f8696a-kube-api-access-5r958\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.205346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.205292 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.205972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.205949 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.207494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.207476 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.268300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.268219 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r958\" (UniqueName: \"kubernetes.io/projected/937a0f9e-1110-490c-9034-3b5c67f8696a-kube-api-access-5r958\") pod \"maas-api-revoke-sub-1bbf95cc-7z5sf\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.309585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.309542 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:44.563894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.563819 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:09:44.742975 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:09:44.741604 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937a0f9e_1110_490c_9034_3b5c67f8696a.slice/crio-c535e086ada641336dc2066f905abc783bea960d7c785a75391be0938723cc38 WatchSource:0}: Error finding container c535e086ada641336dc2066f905abc783bea960d7c785a75391be0938723cc38: Status 404 returned error can't find the container with id c535e086ada641336dc2066f905abc783bea960d7c785a75391be0938723cc38 Nov 03 19:09:44.742975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:44.742842 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf"] Nov 03 19:09:45.612189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:45.612151 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" event={"ID":"937a0f9e-1110-490c-9034-3b5c67f8696a","Type":"ContainerStarted","Data":"6050965e63244d0f0b88b05a525a18a51c6e00e71fcb7d0fd9d86b3c10496ce8"} Nov 03 19:09:45.612595 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:45.612210 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" event={"ID":"937a0f9e-1110-490c-9034-3b5c67f8696a","Type":"ContainerStarted","Data":"c535e086ada641336dc2066f905abc783bea960d7c785a75391be0938723cc38"} Nov 03 19:09:45.732119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:45.732064 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" podStartSLOduration=2.732050036 podStartE2EDuration="2.732050036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:45.728994248 +0000 UTC m=+1915.211990162" watchObservedRunningTime="2026-11-03 19:09:45.732050036 +0000 UTC m=+1915.215045950" Nov 03 19:09:46.622974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:46.622928 4948 generic.go:356] "Generic (PLEG): container finished" podID="937a0f9e-1110-490c-9034-3b5c67f8696a" containerID="6050965e63244d0f0b88b05a525a18a51c6e00e71fcb7d0fd9d86b3c10496ce8" exitCode=0 Nov 03 19:09:46.623411 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:46.623018 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" event={"ID":"937a0f9e-1110-490c-9034-3b5c67f8696a","Type":"ContainerDied","Data":"6050965e63244d0f0b88b05a525a18a51c6e00e71fcb7d0fd9d86b3c10496ce8"} Nov 03 19:09:47.830466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:47.830432 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:47.958494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:47.958436 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937a0f9e-1110-490c-9034-3b5c67f8696a-kube-api-access-5r958\" (UniqueName: \"kubernetes.io/projected/937a0f9e-1110-490c-9034-3b5c67f8696a-kube-api-access-5r958\") pod \"937a0f9e-1110-490c-9034-3b5c67f8696a\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " Nov 03 19:09:47.958722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:47.958508 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-openshift-ca\") pod \"937a0f9e-1110-490c-9034-3b5c67f8696a\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " Nov 03 19:09:47.958722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:47.958558 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-xks-ca\") pod \"937a0f9e-1110-490c-9034-3b5c67f8696a\" (UID: \"937a0f9e-1110-490c-9034-3b5c67f8696a\") " Nov 03 19:09:47.962218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:47.962185 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-xks-ca" pod "937a0f9e-1110-490c-9034-3b5c67f8696a" (UID: "937a0f9e-1110-490c-9034-3b5c67f8696a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:09:47.962394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:47.962307 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937a0f9e-1110-490c-9034-3b5c67f8696a-kube-api-access-5r958" pod "937a0f9e-1110-490c-9034-3b5c67f8696a" (UID: "937a0f9e-1110-490c-9034-3b5c67f8696a"). InnerVolumeSpecName "kube-api-access-5r958". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:09:47.962685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:47.962645 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-openshift-ca" pod "937a0f9e-1110-490c-9034-3b5c67f8696a" (UID: "937a0f9e-1110-490c-9034-3b5c67f8696a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:09:48.061682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:48.061639 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r958\" (UniqueName: \"kubernetes.io/projected/937a0f9e-1110-490c-9034-3b5c67f8696a-kube-api-access-5r958\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:48.061682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:48.061680 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:48.061953 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:48.061731 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a0f9e-1110-490c-9034-3b5c67f8696a-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:09:48.679330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:48.679282 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" event={"ID":"937a0f9e-1110-490c-9034-3b5c67f8696a","Type":"ContainerDied","Data":"c535e086ada641336dc2066f905abc783bea960d7c785a75391be0938723cc38"} Nov 03 19:09:48.679677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:48.679651 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c535e086ada641336dc2066f905abc783bea960d7c785a75391be0938723cc38" Nov 03 19:09:48.680152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:48.680131 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf" Nov 03 19:09:59.470725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:09:59.470667 4948 scope.go:122] "RemoveContainer" containerID="8f1e732747027d6091028d2aad6433c599c09dbd882bfdd674e32bf3ef2738fd" Nov 03 19:10:02.811460 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:10:02.811376 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:10:06.398812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.398772 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8"] Nov 03 19:10:06.399627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.399609 4948 cpu_manager.go:414] "containerMap: removing container" podUID="937a0f9e-1110-490c-9034-3b5c67f8696a" containerName="revoke-keys" Nov 03 19:10:06.399627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.399627 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937a0f9e-1110-490c-9034-3b5c67f8696a" containerName="revoke-keys" Nov 03 19:10:06.399799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.399788 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="937a0f9e-1110-490c-9034-3b5c67f8696a" containerName="revoke-keys" Nov 03 19:10:06.405310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.405292 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.459454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.459361 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:10:06.459454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.459372 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:10:06.489788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.488439 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8"] Nov 03 19:10:06.571725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.545846 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.571725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.546107 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.571725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.546238 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dn9\" (UniqueName: \"kubernetes.io/projected/9d34bca1-6692-4dd0-9696-ca291a40836d-kube-api-access-58dn9\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.710092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.710055 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.710575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.710547 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.710807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.710789 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dn9\" (UniqueName: \"kubernetes.io/projected/9d34bca1-6692-4dd0-9696-ca291a40836d-kube-api-access-58dn9\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.712345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.712294 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.739521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.739390 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.774924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.774891 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dn9\" (UniqueName: \"kubernetes.io/projected/9d34bca1-6692-4dd0-9696-ca291a40836d-kube-api-access-58dn9\") pod \"maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:06.820228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:06.820193 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:07.218350 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:10:07.218317 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d34bca1_6692_4dd0_9696_ca291a40836d.slice/crio-8e598beba89e77da4f0a313da80348242d1582fd8a8ae2e8b65e0efcd7f380f9 WatchSource:0}: Error finding container 8e598beba89e77da4f0a313da80348242d1582fd8a8ae2e8b65e0efcd7f380f9: Status 404 returned error can't find the container with id 8e598beba89e77da4f0a313da80348242d1582fd8a8ae2e8b65e0efcd7f380f9 Nov 03 19:10:07.218669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.218646 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8"] Nov 03 19:10:07.352117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.352071 4948 patch_prober.go:29] interesting pod/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": read tcp 10.132.0.2:58376->10.132.0.193:15021: read: connection reset by peer" start-of-body= Nov 03 19:10:07.352892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.352157 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": read tcp 10.132.0.2:58376->10.132.0.193:15021: read: connection reset by peer" Nov 03 19:10:07.935073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.935036 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" event={"ID":"9d34bca1-6692-4dd0-9696-ca291a40836d","Type":"ContainerStarted","Data":"04f8d30af86c9f3ba412a66fd491ee317e5da3d82640f46a38ecc0874ebec3d0"} Nov 03 19:10:07.935516 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.935094 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" event={"ID":"9d34bca1-6692-4dd0-9696-ca291a40836d","Type":"ContainerStarted","Data":"8e598beba89e77da4f0a313da80348242d1582fd8a8ae2e8b65e0efcd7f380f9"} Nov 03 19:10:07.936883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.936861 4948 generic.go:356] "Generic (PLEG): container finished" podID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerID="5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193" exitCode=0 Nov 03 19:10:07.937037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.936939 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" event={"ID":"44aa37b6-f6ca-4281-a298-e96c0f0417aa","Type":"ContainerDied","Data":"5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193"} Nov 03 19:10:07.937366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.937346 4948 scope.go:122] "RemoveContainer" containerID="5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193" Nov 03 19:10:07.939074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.939045 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:07.939161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.939125 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:07.939203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:07.939171 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:08.059368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:08.059301 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" podStartSLOduration=2.059286227 podStartE2EDuration="2.059286227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:08.056046533 +0000 UTC m=+1937.539042448" watchObservedRunningTime="2026-11-03 19:10:08.059286227 +0000 UTC m=+1937.542282141" Nov 03 19:10:08.947546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:08.947512 4948 generic.go:356] "Generic (PLEG): container finished" podID="9d34bca1-6692-4dd0-9696-ca291a40836d" containerID="04f8d30af86c9f3ba412a66fd491ee317e5da3d82640f46a38ecc0874ebec3d0" exitCode=0 Nov 03 19:10:08.948000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:08.947590 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" event={"ID":"9d34bca1-6692-4dd0-9696-ca291a40836d","Type":"ContainerDied","Data":"04f8d30af86c9f3ba412a66fd491ee317e5da3d82640f46a38ecc0874ebec3d0"} Nov 03 19:10:08.949883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:08.949853 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" event={"ID":"44aa37b6-f6ca-4281-a298-e96c0f0417aa","Type":"ContainerStarted","Data":"f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49"} Nov 03 19:10:09.646395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:09.646359 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:10:09.646617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:09.646565 4948 patch_prober.go:29] interesting pod/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 19:10:09.646695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:09.646619 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 19:10:10.428662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.428637 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:10.529326 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.529232 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-openshift-ca\") pod \"9d34bca1-6692-4dd0-9696-ca291a40836d\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " Nov 03 19:10:10.529326 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.529296 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-xks-ca\") pod \"9d34bca1-6692-4dd0-9696-ca291a40836d\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " Nov 03 19:10:10.529584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.529338 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d34bca1-6692-4dd0-9696-ca291a40836d-kube-api-access-58dn9\" (UniqueName: \"kubernetes.io/projected/9d34bca1-6692-4dd0-9696-ca291a40836d-kube-api-access-58dn9\") pod \"9d34bca1-6692-4dd0-9696-ca291a40836d\" (UID: \"9d34bca1-6692-4dd0-9696-ca291a40836d\") " Nov 03 19:10:10.529687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.529658 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-openshift-ca" pod "9d34bca1-6692-4dd0-9696-ca291a40836d" (UID: "9d34bca1-6692-4dd0-9696-ca291a40836d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:10:10.529875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.529857 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:10.531465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.531440 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-xks-ca" pod "9d34bca1-6692-4dd0-9696-ca291a40836d" (UID: "9d34bca1-6692-4dd0-9696-ca291a40836d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:10:10.531546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.531472 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d34bca1-6692-4dd0-9696-ca291a40836d-kube-api-access-58dn9" pod "9d34bca1-6692-4dd0-9696-ca291a40836d" (UID: "9d34bca1-6692-4dd0-9696-ca291a40836d"). InnerVolumeSpecName "kube-api-access-58dn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:10:10.632026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.631992 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34bca1-6692-4dd0-9696-ca291a40836d-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:10.632026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.632024 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58dn9\" (UniqueName: \"kubernetes.io/projected/9d34bca1-6692-4dd0-9696-ca291a40836d-kube-api-access-58dn9\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:10.647022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.646988 4948 patch_prober.go:29] interesting pod/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 19:10:10.647136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.647056 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 19:10:10.909777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.909727 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d"] Nov 03 19:10:10.910971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.910951 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9d34bca1-6692-4dd0-9696-ca291a40836d" containerName="revoke-keys" Nov 03 19:10:10.911056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.910974 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34bca1-6692-4dd0-9696-ca291a40836d" containerName="revoke-keys" Nov 03 19:10:10.911141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.911129 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34bca1-6692-4dd0-9696-ca291a40836d" containerName="revoke-keys" Nov 03 19:10:10.915000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.914978 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:10.992988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:10.991672 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-97e020-openshift-default-dockercfg-htkrg\"" Nov 03 19:10:11.010152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.010112 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" event={"ID":"9d34bca1-6692-4dd0-9696-ca291a40836d","Type":"ContainerDied","Data":"8e598beba89e77da4f0a313da80348242d1582fd8a8ae2e8b65e0efcd7f380f9"} Nov 03 19:10:11.010324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.010165 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8" Nov 03 19:10:11.010324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.010177 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e598beba89e77da4f0a313da80348242d1582fd8a8ae2e8b65e0efcd7f380f9" Nov 03 19:10:11.038676 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.038636 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0263c0-0903-4b42-94e5-2d65d5823087-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.038896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.038791 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.038896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.038827 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.038896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.038868 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.039048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.038928 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px4jr\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-kube-api-access-px4jr\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.039125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.039097 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.039254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.039226 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.039310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.039274 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.039385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.039361 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.141767 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.141827 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.141889 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.141922 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px4jr\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-kube-api-access-px4jr\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.141985 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.142098 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.142128 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.142200 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.142309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.142303 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0263c0-0903-4b42-94e5-2d65d5823087-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.143327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.143292 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0263c0-0903-4b42-94e5-2d65d5823087-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.144153 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.144123 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.157530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.146800 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.157530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.147430 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.157530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.147692 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.268719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.229915 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d"] Nov 03 19:10:11.376316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.376278 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.397768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.391689 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.397768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.392467 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.397768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.394105 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px4jr\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-kube-api-access-px4jr\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.532753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.532716 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:11.646301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.646265 4948 patch_prober.go:29] interesting pod/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 19:10:11.646467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:11.646326 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 19:10:12.016190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:12.016152 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d"] Nov 03 19:10:12.018602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:12.018565 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:12.018726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:12.018651 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:12.018726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:12.018693 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:12.646811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:12.646774 4948 patch_prober.go:29] interesting pod/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 19:10:12.647310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:12.646838 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 19:10:13.033134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:13.033092 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" event={"ID":"5d0263c0-0903-4b42-94e5-2d65d5823087","Type":"ContainerStarted","Data":"a45a71dc16ec3ddf522bc0d2f8d16982d7c7945f602eb456b4edd527273cc3f5"} Nov 03 19:10:13.033320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:13.033156 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" event={"ID":"5d0263c0-0903-4b42-94e5-2d65d5823087","Type":"ContainerStarted","Data":"23c029030dc107c3b959ff9af4ca92aa40b9cde3b695c5a10e74865ebb77c816"} Nov 03 19:10:13.279726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:13.279489 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" podStartSLOduration=3.279452307 podStartE2EDuration="3.279452307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:13.205829713 +0000 UTC m=+1942.688825630" watchObservedRunningTime="2026-11-03 19:10:13.279452307 +0000 UTC m=+1942.762448222" Nov 03 19:10:13.533208 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:13.533172 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:13.539575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:13.539544 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:13.646279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:13.646244 4948 patch_prober.go:29] interesting pod/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 19:10:13.646480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:13.646305 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 19:10:14.044363 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:14.044327 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:14.045497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:14.045476 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:10:14.767233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:14.763720 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:10:14.767233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:14.765279 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:10:14.767233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:14.766218 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:10:17.942774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:17.942735 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9"] Nov 03 19:10:18.059083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:18.059046 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb"] Nov 03 19:10:22.643467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.643432 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc"] Nov 03 19:10:22.648035 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.647998 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:22.725756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.716481 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-97e020\"" Nov 03 19:10:22.738145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.729655 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc"] Nov 03 19:10:22.802745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.793986 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-97e020-dockercfg-6dg4v\"" Nov 03 19:10:22.886726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.880109 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66069f67-85da-41fc-8868-2ddec44a172a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:22.886726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.880359 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwck\" (UniqueName: \"kubernetes.io/projected/66069f67-85da-41fc-8868-2ddec44a172a-kube-api-access-ncwck\") pod \"payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:22.946779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.944817 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" podUID="ccc02de7-a8a0-4c12-b809-11034c3ca45e" containerName="payload-processing" containerID="cri-o://5969b6114e21fb44073e6f075b4fb3a26438c01ed3b52abf881c10ad7d518fe2" gracePeriod=25 Nov 03 19:10:22.986643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.986386 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66069f67-85da-41fc-8868-2ddec44a172a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:22.986643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.986593 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwck\" (UniqueName: \"kubernetes.io/projected/66069f67-85da-41fc-8868-2ddec44a172a-kube-api-access-ncwck\") pod \"payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:22.988414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.988329 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8"] Nov 03 19:10:22.988414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.988359 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66069f67-85da-41fc-8868-2ddec44a172a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:22.998359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:22.998327 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.013743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.013688 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8"] Nov 03 19:10:23.049205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.049174 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwck\" (UniqueName: \"kubernetes.io/projected/66069f67-85da-41fc-8868-2ddec44a172a-kube-api-access-ncwck\") pod \"payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:23.064772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.059353 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" podUID="44530f1b-e75a-4856-8b82-c7cc85440525" containerName="payload-pre-processing" containerID="cri-o://e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6" gracePeriod=25 Nov 03 19:10:23.087726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.087674 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv64g\" (UniqueName: \"kubernetes.io/projected/6534ac32-568f-4564-9bd8-3e1b9d190fdc-kube-api-access-tv64g\") pod \"payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.088338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.088299 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534ac32-568f-4564-9bd8-3e1b9d190fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.134834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.134801 4948 generic.go:356] "Generic (PLEG): container finished" podID="ccc02de7-a8a0-4c12-b809-11034c3ca45e" containerID="5969b6114e21fb44073e6f075b4fb3a26438c01ed3b52abf881c10ad7d518fe2" exitCode=0 Nov 03 19:10:23.134995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.134886 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" event={"ID":"ccc02de7-a8a0-4c12-b809-11034c3ca45e","Type":"ContainerDied","Data":"5969b6114e21fb44073e6f075b4fb3a26438c01ed3b52abf881c10ad7d518fe2"} Nov 03 19:10:23.190126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.190089 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534ac32-568f-4564-9bd8-3e1b9d190fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.190298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.190277 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv64g\" (UniqueName: \"kubernetes.io/projected/6534ac32-568f-4564-9bd8-3e1b9d190fdc-kube-api-access-tv64g\") pod \"payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.191143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.191116 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534ac32-568f-4564-9bd8-3e1b9d190fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.254619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.253180 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv64g\" (UniqueName: \"kubernetes.io/projected/6534ac32-568f-4564-9bd8-3e1b9d190fdc-kube-api-access-tv64g\") pod \"payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.266830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.263984 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:23.352239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.352204 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:10:23.405072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.405045 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:10:23.463739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.463316 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:23.494869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.494821 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44530f1b-e75a-4856-8b82-c7cc85440525-plugins-config-volume" pod "44530f1b-e75a-4856-8b82-c7cc85440525" (UID: "44530f1b-e75a-4856-8b82-c7cc85440525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:10:23.495092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.494906 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44530f1b-e75a-4856-8b82-c7cc85440525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44530f1b-e75a-4856-8b82-c7cc85440525-plugins-config-volume\") pod \"44530f1b-e75a-4856-8b82-c7cc85440525\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " Nov 03 19:10:23.495978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.495099 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc02de7-a8a0-4c12-b809-11034c3ca45e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc02de7-a8a0-4c12-b809-11034c3ca45e-plugins-config-volume\") pod \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " Nov 03 19:10:23.495978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.495635 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc02de7-a8a0-4c12-b809-11034c3ca45e-plugins-config-volume" pod "ccc02de7-a8a0-4c12-b809-11034c3ca45e" (UID: "ccc02de7-a8a0-4c12-b809-11034c3ca45e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:10:23.495978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.495780 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc02de7-a8a0-4c12-b809-11034c3ca45e-kube-api-access-74b98\" (UniqueName: \"kubernetes.io/projected/ccc02de7-a8a0-4c12-b809-11034c3ca45e-kube-api-access-74b98\") pod \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\" (UID: \"ccc02de7-a8a0-4c12-b809-11034c3ca45e\") " Nov 03 19:10:23.496574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.496550 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44530f1b-e75a-4856-8b82-c7cc85440525-kube-api-access-vzl75\" (UniqueName: \"kubernetes.io/projected/44530f1b-e75a-4856-8b82-c7cc85440525-kube-api-access-vzl75\") pod \"44530f1b-e75a-4856-8b82-c7cc85440525\" (UID: \"44530f1b-e75a-4856-8b82-c7cc85440525\") " Nov 03 19:10:23.497515 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.497492 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44530f1b-e75a-4856-8b82-c7cc85440525-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:23.497626 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.497519 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc02de7-a8a0-4c12-b809-11034c3ca45e-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:23.498627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.498586 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc02de7-a8a0-4c12-b809-11034c3ca45e-kube-api-access-74b98" pod "ccc02de7-a8a0-4c12-b809-11034c3ca45e" (UID: "ccc02de7-a8a0-4c12-b809-11034c3ca45e"). InnerVolumeSpecName "kube-api-access-74b98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:10:23.499519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.499486 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44530f1b-e75a-4856-8b82-c7cc85440525-kube-api-access-vzl75" pod "44530f1b-e75a-4856-8b82-c7cc85440525" (UID: "44530f1b-e75a-4856-8b82-c7cc85440525"). InnerVolumeSpecName "kube-api-access-vzl75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:10:23.599480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.599416 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74b98\" (UniqueName: \"kubernetes.io/projected/ccc02de7-a8a0-4c12-b809-11034c3ca45e-kube-api-access-74b98\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:23.599480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.599453 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzl75\" (UniqueName: \"kubernetes.io/projected/44530f1b-e75a-4856-8b82-c7cc85440525-kube-api-access-vzl75\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:23.731366 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:10:23.731326 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66069f67_85da_41fc_8868_2ddec44a172a.slice/crio-6728c0e9d2e6ce58dbbd523bdfd993084dc6f43430621dc073ebe321b53f1016 WatchSource:0}: Error finding container 6728c0e9d2e6ce58dbbd523bdfd993084dc6f43430621dc073ebe321b53f1016: Status 404 returned error can't find the container with id 6728c0e9d2e6ce58dbbd523bdfd993084dc6f43430621dc073ebe321b53f1016 Nov 03 19:10:23.732670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.732645 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc"] Nov 03 19:10:23.866343 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:10:23.866310 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6534ac32_568f_4564_9bd8_3e1b9d190fdc.slice/crio-4824cd6e59c6117d722e6ecdbf1e3d7b873fb185a4a9025a097cac655d05da5e WatchSource:0}: Error finding container 4824cd6e59c6117d722e6ecdbf1e3d7b873fb185a4a9025a097cac655d05da5e: Status 404 returned error can't find the container with id 4824cd6e59c6117d722e6ecdbf1e3d7b873fb185a4a9025a097cac655d05da5e Nov 03 19:10:23.867186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:23.867154 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8"] Nov 03 19:10:24.146645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.146607 4948 generic.go:356] "Generic (PLEG): container finished" podID="44530f1b-e75a-4856-8b82-c7cc85440525" containerID="e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6" exitCode=0 Nov 03 19:10:24.146831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.146666 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" Nov 03 19:10:24.146831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.146684 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" event={"ID":"44530f1b-e75a-4856-8b82-c7cc85440525","Type":"ContainerDied","Data":"e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6"} Nov 03 19:10:24.146831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.146757 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb" event={"ID":"44530f1b-e75a-4856-8b82-c7cc85440525","Type":"ContainerDied","Data":"12a0c98508f98e52355c1614ef53bedd48ef93fa466f3291a4650076979ba069"} Nov 03 19:10:24.146831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.146789 4948 scope.go:122] "RemoveContainer" containerID="e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6" Nov 03 19:10:24.148843 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.148815 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" event={"ID":"66069f67-85da-41fc-8868-2ddec44a172a","Type":"ContainerStarted","Data":"8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e"} Nov 03 19:10:24.148995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.148861 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" event={"ID":"66069f67-85da-41fc-8868-2ddec44a172a","Type":"ContainerStarted","Data":"6728c0e9d2e6ce58dbbd523bdfd993084dc6f43430621dc073ebe321b53f1016"} Nov 03 19:10:24.148995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.148930 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:24.151003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.150976 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" event={"ID":"6534ac32-568f-4564-9bd8-3e1b9d190fdc","Type":"ContainerStarted","Data":"414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d"} Nov 03 19:10:24.151124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.151029 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" event={"ID":"6534ac32-568f-4564-9bd8-3e1b9d190fdc","Type":"ContainerStarted","Data":"4824cd6e59c6117d722e6ecdbf1e3d7b873fb185a4a9025a097cac655d05da5e"} Nov 03 19:10:24.151124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.151076 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:24.152590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.152570 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" event={"ID":"ccc02de7-a8a0-4c12-b809-11034c3ca45e","Type":"ContainerDied","Data":"3177183bdf165cfcb7bc14b60f731ab56bc3481ee7e6e2ec781b3e1fd9eccca9"} Nov 03 19:10:24.152709 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.152615 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9" Nov 03 19:10:24.160758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.160740 4948 scope.go:122] "RemoveContainer" containerID="e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6" Nov 03 19:10:24.161077 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:10:24.161052 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6\": container with ID starting with e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6 not found: ID does not exist" containerID="e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6" Nov 03 19:10:24.161198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.161086 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6"} err="failed to get container status \"e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6\": rpc error: code = NotFound desc = could not find container \"e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6\": container with ID starting with e2642475acca26d4f0c66e15e4461a67f2c37af10769f494b66fc19fd1f50fc6 not found: ID does not exist" Nov 03 19:10:24.161198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.161104 4948 scope.go:122] "RemoveContainer" containerID="5969b6114e21fb44073e6f075b4fb3a26438c01ed3b52abf881c10ad7d518fe2" Nov 03 19:10:24.317578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.317495 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9"] Nov 03 19:10:24.322113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.322091 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5ac14ff6-cf94fcf6c-75dv9"] Nov 03 19:10:24.435358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.435330 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb"] Nov 03 19:10:24.443274 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.443243 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5ac14ff6-7679584699-qqfnb"] Nov 03 19:10:24.549262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.549203 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" podStartSLOduration=2.549187424 podStartE2EDuration="2.549187424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:24.547900332 +0000 UTC m=+1954.030896248" watchObservedRunningTime="2026-11-03 19:10:24.549187424 +0000 UTC m=+1954.032183369" Nov 03 19:10:24.713719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.699321 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" podStartSLOduration=2.699298776 podStartE2EDuration="2.699298776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:24.665641869 +0000 UTC m=+1954.148637786" watchObservedRunningTime="2026-11-03 19:10:24.699298776 +0000 UTC m=+1954.182294693" Nov 03 19:10:24.761074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.761039 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44530f1b-e75a-4856-8b82-c7cc85440525" path="/var/lib/kubelet/pods/44530f1b-e75a-4856-8b82-c7cc85440525/volumes" Nov 03 19:10:24.761566 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:24.761549 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc02de7-a8a0-4c12-b809-11034c3ca45e" path="/var/lib/kubelet/pods/ccc02de7-a8a0-4c12-b809-11034c3ca45e/volumes" Nov 03 19:10:32.136977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.136940 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk"] Nov 03 19:10:32.137823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.137802 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ccc02de7-a8a0-4c12-b809-11034c3ca45e" containerName="payload-processing" Nov 03 19:10:32.137823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.137822 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc02de7-a8a0-4c12-b809-11034c3ca45e" containerName="payload-processing" Nov 03 19:10:32.137973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.137852 4948 cpu_manager.go:414] "containerMap: removing container" podUID="44530f1b-e75a-4856-8b82-c7cc85440525" containerName="payload-pre-processing" Nov 03 19:10:32.137973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.137857 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44530f1b-e75a-4856-8b82-c7cc85440525" containerName="payload-pre-processing" Nov 03 19:10:32.137973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.137970 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="44530f1b-e75a-4856-8b82-c7cc85440525" containerName="payload-pre-processing" Nov 03 19:10:32.138071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.137987 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc02de7-a8a0-4c12-b809-11034c3ca45e" containerName="payload-processing" Nov 03 19:10:32.141628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.141606 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.198479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.198433 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk"] Nov 03 19:10:32.212936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.212898 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-metrics-certs\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.213084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.212945 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jcxw\" (UniqueName: \"kubernetes.io/projected/bcefc690-86a7-4d07-888d-682eb583ce92-kube-api-access-4jcxw\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.213084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.213062 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.213183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.213135 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc690-86a7-4d07-888d-682eb583ce92-openshift-service-ca\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.242039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.242008 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e1f8f8-serving-cert\"" Nov 03 19:10:32.315363 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.315322 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-metrics-certs\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.315568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.315371 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jcxw\" (UniqueName: \"kubernetes.io/projected/bcefc690-86a7-4d07-888d-682eb583ce92-kube-api-access-4jcxw\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.315568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.315420 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.315568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.315495 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc690-86a7-4d07-888d-682eb583ce92-openshift-service-ca\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.315846 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:10:32.315617 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-serving-cert: secret "maas-api-e2e-api-a-e1f8f8-serving-cert" not found Nov 03 19:10:32.315846 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:10:32.315749 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls podName:bcefc690-86a7-4d07-888d-682eb583ce92 nodeName:}" failed. No retries permitted until 2026-11-03 19:10:32.815681447 +0000 UTC m=+1962.298677352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls") pod "maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" (UID: "bcefc690-86a7-4d07-888d-682eb583ce92") : secret "maas-api-e2e-api-a-e1f8f8-serving-cert" not found Nov 03 19:10:32.316277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.316255 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc690-86a7-4d07-888d-682eb583ce92-openshift-service-ca\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.318337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.318306 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-metrics-certs\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.502056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.496821 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jcxw\" (UniqueName: \"kubernetes.io/projected/bcefc690-86a7-4d07-888d-682eb583ce92-kube-api-access-4jcxw\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.761963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.761891 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg"] Nov 03 19:10:32.765759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.765741 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.849523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849472 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-socket\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.849728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849558 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-certs\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.849728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849619 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcpj8\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-kube-api-access-pcpj8\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.849728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849670 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-credential-socket\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.849900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849794 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.849900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849865 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc16153-2d17-4bb4-a11b-efb0355158be-istiod-ca-cert\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.850012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849900 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-envoy\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.850012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.849937 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-istio-token\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.850122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.850014 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc16153-2d17-4bb4-a11b-efb0355158be-istio-podinfo\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.850122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.850071 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-data\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.863860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.863803 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls\") pod \"maas-api-e2e-api-a-e1f8f8-f49765459-65qrk\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:32.866474 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:10:32.866394 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:10:32.885247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.885195 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a93912-openshift-default-dockercfg-275pc\"" Nov 03 19:10:32.983742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.973693 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-envoy\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.984572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.984194 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-istio-token\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.988047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.987868 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc16153-2d17-4bb4-a11b-efb0355158be-istio-podinfo\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:32.996272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:32.996232 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-data\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.059980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.001098 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-data\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.081734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.078376 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:33.090737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.090639 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-istio-token\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.100963 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-socket\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.101042 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-certs\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.101113 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcpj8\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-kube-api-access-pcpj8\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.101171 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-credential-socket\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.101293 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc16153-2d17-4bb4-a11b-efb0355158be-istiod-ca-cert\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.102112 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc16153-2d17-4bb4-a11b-efb0355158be-istiod-ca-cert\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.102355 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-socket\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.102595 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-certs\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.105514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.104952 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-credential-socket\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.249312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.232657 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg"] Nov 03 19:10:33.249312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.235280 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-envoy\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.249312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.237275 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcpj8\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-kube-api-access-pcpj8\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.252640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.252606 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc16153-2d17-4bb4-a11b-efb0355158be-istio-podinfo\") pod \"e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.392032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.391928 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:33.623096 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:10:33.623061 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcefc690_86a7_4d07_888d_682eb583ce92.slice/crio-52a95a0670f9c496363744d8577e059221ed6d48ab361bd490d6532913b796b9 WatchSource:0}: Error finding container 52a95a0670f9c496363744d8577e059221ed6d48ab361bd490d6532913b796b9: Status 404 returned error can't find the container with id 52a95a0670f9c496363744d8577e059221ed6d48ab361bd490d6532913b796b9 Nov 03 19:10:33.624139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.624114 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk"] Nov 03 19:10:33.787551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.787517 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg"] Nov 03 19:10:33.788243 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:10:33.788217 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc16153_2d17_4bb4_a11b_efb0355158be.slice/crio-dc44bfe2591d6a01105fe86a21ea591d15db3bf10df130fcae38d820bf81a922 WatchSource:0}: Error finding container dc44bfe2591d6a01105fe86a21ea591d15db3bf10df130fcae38d820bf81a922: Status 404 returned error can't find the container with id dc44bfe2591d6a01105fe86a21ea591d15db3bf10df130fcae38d820bf81a922 Nov 03 19:10:33.792069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.792033 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:33.792219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.792119 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:33.792219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:33.792160 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:34.312724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:34.305746 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" event={"ID":"acc16153-2d17-4bb4-a11b-efb0355158be","Type":"ContainerStarted","Data":"f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce"} Nov 03 19:10:34.312724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:34.305813 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" event={"ID":"acc16153-2d17-4bb4-a11b-efb0355158be","Type":"ContainerStarted","Data":"dc44bfe2591d6a01105fe86a21ea591d15db3bf10df130fcae38d820bf81a922"} Nov 03 19:10:34.324725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:34.320008 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" event={"ID":"bcefc690-86a7-4d07-888d-682eb583ce92","Type":"ContainerStarted","Data":"52a95a0670f9c496363744d8577e059221ed6d48ab361bd490d6532913b796b9"} Nov 03 19:10:34.392952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:34.392916 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:34.401447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:34.401413 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:34.433470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:34.433407 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podStartSLOduration=2.4333882620000002 podStartE2EDuration="2.433388262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:34.430072859 +0000 UTC m=+1963.913068785" watchObservedRunningTime="2026-11-03 19:10:34.433388262 +0000 UTC m=+1963.916384175" Nov 03 19:10:35.241480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:35.212894 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:10:35.241480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:35.214426 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:10:35.332869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:35.332827 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" event={"ID":"bcefc690-86a7-4d07-888d-682eb583ce92","Type":"ContainerStarted","Data":"0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c"} Nov 03 19:10:35.333382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:35.333059 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:35.334323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:35.334297 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:10:35.466365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:35.466300 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" podStartSLOduration=2.608643752 podStartE2EDuration="3.466279016s" totalImagesPullingTime="857.635264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:35.462763221 +0000 UTC m=+1964.945759136" watchObservedRunningTime="2026-11-03 19:10:35.466279016 +0000 UTC m=+1964.949274931" Nov 03 19:10:36.350726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:36.344612 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:37.353834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:37.353797 4948 generic.go:356] "Generic (PLEG): container finished" podID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerID="8dc21d0b922a2231a1e793cae12e19ad49ad94ad2f0296117eb0cacd59b1b9db" exitCode=0 Nov 03 19:10:37.354419 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:37.353880 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" event={"ID":"39803e8e-ba6f-449a-9d53-116d0147e05d","Type":"ContainerDied","Data":"8dc21d0b922a2231a1e793cae12e19ad49ad94ad2f0296117eb0cacd59b1b9db"} Nov 03 19:10:37.354518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:37.354460 4948 scope.go:122] "RemoveContainer" containerID="8dc21d0b922a2231a1e793cae12e19ad49ad94ad2f0296117eb0cacd59b1b9db" Nov 03 19:10:37.356327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:37.356295 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:37.356450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:37.356367 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:37.356450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:37.356408 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:38.379722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:38.378998 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" event={"ID":"39803e8e-ba6f-449a-9d53-116d0147e05d","Type":"ContainerStarted","Data":"ea555dd5988dfa441f6828aa68093fc36eecdc00030d2c6e2524db91dfe5a12a"} Nov 03 19:10:39.375112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:39.374135 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:10:39.375112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:39.374278 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 19:10:39.375112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:39.374325 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 19:10:40.377723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:40.374129 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 19:10:40.377723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:40.374199 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 19:10:41.406724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.386846 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 19:10:41.406724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.386911 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 19:10:41.504507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.504471 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:10:41.560507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.558210 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw"] Nov 03 19:10:41.586725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.583563 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw"] Nov 03 19:10:41.586725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.583719 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:41.823723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.810066 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e59596-cda8-46bd-a896-4dd507707ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:41.823723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.810209 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbch\" (UniqueName: \"kubernetes.io/projected/c3e59596-cda8-46bd-a896-4dd507707ec9-kube-api-access-sdbch\") pod \"payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:41.980724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.912967 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e59596-cda8-46bd-a896-4dd507707ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:42.003680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.941358 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a93912\"" Nov 03 19:10:42.003680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.947363 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e59596-cda8-46bd-a896-4dd507707ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:42.003680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.996746 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r"] Nov 03 19:10:42.003680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:41.997998 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbch\" (UniqueName: \"kubernetes.io/projected/c3e59596-cda8-46bd-a896-4dd507707ec9-kube-api-access-sdbch\") pod \"payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:42.023882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.022833 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r"] Nov 03 19:10:42.023882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.022952 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.086783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.083251 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a93912-dockercfg-5589v\"" Nov 03 19:10:42.091998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.091949 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbch\" (UniqueName: \"kubernetes.io/projected/c3e59596-cda8-46bd-a896-4dd507707ec9-kube-api-access-sdbch\") pod \"payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:42.100199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.100159 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cf7280-59ea-4c15-b677-de5cf836f300-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.100473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.100458 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6rh\" (UniqueName: \"kubernetes.io/projected/10cf7280-59ea-4c15-b677-de5cf836f300-kube-api-access-nc6rh\") pod \"payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.201780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.201747 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6rh\" (UniqueName: \"kubernetes.io/projected/10cf7280-59ea-4c15-b677-de5cf836f300-kube-api-access-nc6rh\") pod \"payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.201917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.201808 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cf7280-59ea-4c15-b677-de5cf836f300-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.202527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.202506 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cf7280-59ea-4c15-b677-de5cf836f300-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.273730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.272911 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:42.288631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.288562 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6rh\" (UniqueName: \"kubernetes.io/projected/10cf7280-59ea-4c15-b677-de5cf836f300-kube-api-access-nc6rh\") pod \"payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.343548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.343464 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:42.398142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.385529 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 19:10:42.398142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.385584 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 19:10:42.398142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.392307 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:10:42.693487 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:10:42.693457 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e59596_cda8_46bd_a896_4dd507707ec9.slice/crio-9b01360eb8deebc3da2e9e53ebfec5bd7281d59363276c85348fe76460f5c32b WatchSource:0}: Error finding container 9b01360eb8deebc3da2e9e53ebfec5bd7281d59363276c85348fe76460f5c32b: Status 404 returned error can't find the container with id 9b01360eb8deebc3da2e9e53ebfec5bd7281d59363276c85348fe76460f5c32b Nov 03 19:10:42.694292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.694265 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw"] Nov 03 19:10:42.745126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.745095 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" event={"ID":"c3e59596-cda8-46bd-a896-4dd507707ec9","Type":"ContainerStarted","Data":"9b01360eb8deebc3da2e9e53ebfec5bd7281d59363276c85348fe76460f5c32b"} Nov 03 19:10:42.812046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:42.812020 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r"] Nov 03 19:10:43.375778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:43.375079 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 19:10:43.375778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:43.375146 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 19:10:43.868855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:43.864090 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" event={"ID":"10cf7280-59ea-4c15-b677-de5cf836f300","Type":"ContainerStarted","Data":"34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f"} Nov 03 19:10:43.868855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:43.864157 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" event={"ID":"10cf7280-59ea-4c15-b677-de5cf836f300","Type":"ContainerStarted","Data":"055101c2b16d20bfabed0535678f7e724ac13f4214ed40b22217c5330b1a3363"} Nov 03 19:10:43.868855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:43.865322 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:43.882475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:43.875003 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" event={"ID":"c3e59596-cda8-46bd-a896-4dd507707ec9","Type":"ContainerStarted","Data":"6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45"} Nov 03 19:10:43.886740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:43.883903 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:44.019726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:44.019016 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" podStartSLOduration=3.018992875 podStartE2EDuration="3.018992875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:43.988296809 +0000 UTC m=+1973.471292727" watchObservedRunningTime="2026-11-03 19:10:44.018992875 +0000 UTC m=+1973.501988790" Nov 03 19:10:44.140794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:44.137675 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" podStartSLOduration=3.1376496449999998 podStartE2EDuration="3.137649645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:44.129208939 +0000 UTC m=+1973.612204915" watchObservedRunningTime="2026-11-03 19:10:44.137649645 +0000 UTC m=+1973.620645560" Nov 03 19:10:44.374726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:44.371115 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 19:10:44.374726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:44.371178 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 19:10:45.043724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.039822 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 03 19:10:45.069723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.065828 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4w4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:10:45.069723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.065904 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:45.201511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.201456 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 03 19:10:45.275974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.275914 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:45.394785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.391995 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:10:45.422954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.422924 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:10:45.526723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.525793 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:45.835728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.786426 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:45.894529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.845284 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:45.894529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:45.845623 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 19:10:46.044877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.044818 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:46.050944 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.048522 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:46.057772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.051227 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:10:46.206907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.205759 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:46.206907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.206237 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:10:46.236740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.233116 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:10:46.314772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.295915 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:10:46.314772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.296368 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:10:46.326912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.326305 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:10:46.344102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:46.335234 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:10:54.027235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.027199 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn"] Nov 03 19:10:54.053719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.050554 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.165068 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.165024 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-efd81c-openshift-default-dockercfg-tmg8s\"" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187359 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-token\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187464 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-credential-socket\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187557 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjf7n\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-kube-api-access-kjf7n\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187584 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-envoy\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187635 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-data\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187669 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-socket\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187694 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-podinfo\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187824 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f8681-850f-4ce3-baa4-368173ab3179-istiod-ca-cert\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.201874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.187861 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-certs\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.297898 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-token\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298027 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-credential-socket\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298139 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjf7n\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-kube-api-access-kjf7n\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298170 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-envoy\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298255 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-data\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298305 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-socket\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298336 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-podinfo\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298481 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f8681-850f-4ce3-baa4-368173ab3179-istiod-ca-cert\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.298530 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-certs\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.299152 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-certs\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.299775 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-credential-socket\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.303505 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-envoy\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.303832 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-data\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.338790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.304097 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-socket\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.341087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.340304 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-podinfo\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.341087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.341025 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f8681-850f-4ce3-baa4-368173ab3179-istiod-ca-cert\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.367916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.366013 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn"] Nov 03 19:10:54.421177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.421110 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjf7n\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-kube-api-access-kjf7n\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.427664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.427588 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-token\") pod \"e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:54.712634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:54.712599 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:55.151590 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:10:55.151555 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4f8681_850f_4ce3_baa4_368173ab3179.slice/crio-0538da0abbad9233f3c3c5dd05798ffb31d3af623fa8b47df1bb437520ff73fa WatchSource:0}: Error finding container 0538da0abbad9233f3c3c5dd05798ffb31d3af623fa8b47df1bb437520ff73fa: Status 404 returned error can't find the container with id 0538da0abbad9233f3c3c5dd05798ffb31d3af623fa8b47df1bb437520ff73fa Nov 03 19:10:55.152593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.152567 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn"] Nov 03 19:10:55.154652 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.154618 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:55.154750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.154683 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:55.154750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.154731 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:10:55.424082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.423979 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" event={"ID":"ee4f8681-850f-4ce3-baa4-368173ab3179","Type":"ContainerStarted","Data":"c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762"} Nov 03 19:10:55.424082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.424076 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" event={"ID":"ee4f8681-850f-4ce3-baa4-368173ab3179","Type":"ContainerStarted","Data":"0538da0abbad9233f3c3c5dd05798ffb31d3af623fa8b47df1bb437520ff73fa"} Nov 03 19:10:55.547433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.547377 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podStartSLOduration=2.547361971 podStartE2EDuration="2.547361971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:55.545883922 +0000 UTC m=+1985.028879837" watchObservedRunningTime="2026-11-03 19:10:55.547361971 +0000 UTC m=+1985.030357885" Nov 03 19:10:55.720728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:55.714046 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:56.140497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:56.140401 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:10:56.140913 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:56.140890 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:10:56.719958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:56.719932 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:57.446001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:57.445963 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:57.447089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:57.447066 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:10:59.509321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:59.509289 4948 scope.go:122] "RemoveContainer" containerID="96423b30a93111d9cacf217466755bf20accdcb126640457d8dcdd041fbc7761" Nov 03 19:10:59.521252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:59.521232 4948 scope.go:122] "RemoveContainer" containerID="3fac942a7054f8849937ee718331fcad60a0306585ad7dfefaa1611772db7b83" Nov 03 19:10:59.532949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:10:59.532926 4948 scope.go:122] "RemoveContainer" containerID="aa4f44ad9f91d34c8200fc8117d93c4f79d1ee0373914baf8517f2b8da4de5cf" Nov 03 19:11:02.796652 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:11:02.796607 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:11:04.630724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.624054 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f"] Nov 03 19:11:04.663723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.652282 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:04.808085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.789937 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:04.808085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.790141 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljqmh\" (UniqueName: \"kubernetes.io/projected/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-kube-api-access-ljqmh\") pod \"payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:04.871047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.871007 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-efd81c-dockercfg-w2l66\"" Nov 03 19:11:04.871256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.871136 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-efd81c\"" Nov 03 19:11:04.899460 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.899361 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:04.899650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.899549 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljqmh\" (UniqueName: \"kubernetes.io/projected/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-kube-api-access-ljqmh\") pod \"payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:04.913229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.900957 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:04.990815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:04.990767 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljqmh\" (UniqueName: \"kubernetes.io/projected/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-kube-api-access-ljqmh\") pod \"payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:05.004924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:05.004329 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:05.012330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:05.012295 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f"] Nov 03 19:11:05.438915 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:11:05.438868 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbfb1bf_939d_4ef0_a851_f9e35ed780b8.slice/crio-572553d29558983762ec22fb57802f032ebee9dc9f25a1d9564b7338e0b67afe WatchSource:0}: Error finding container 572553d29558983762ec22fb57802f032ebee9dc9f25a1d9564b7338e0b67afe: Status 404 returned error can't find the container with id 572553d29558983762ec22fb57802f032ebee9dc9f25a1d9564b7338e0b67afe Nov 03 19:11:05.441851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:05.441824 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f"] Nov 03 19:11:05.622593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:05.622561 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" event={"ID":"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8","Type":"ContainerStarted","Data":"3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5"} Nov 03 19:11:05.622734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:05.622622 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" event={"ID":"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8","Type":"ContainerStarted","Data":"572553d29558983762ec22fb57802f032ebee9dc9f25a1d9564b7338e0b67afe"} Nov 03 19:11:05.622734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:05.622709 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:05.788068 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:05.787952 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" podStartSLOduration=1.787932267 podStartE2EDuration="1.787932267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:05.787213124 +0000 UTC m=+1995.270209038" watchObservedRunningTime="2026-11-03 19:11:05.787932267 +0000 UTC m=+1995.270928183" Nov 03 19:11:07.596782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.595995 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5"] Nov 03 19:11:07.600196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.600169 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.655731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.655529 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-042f66-openshift-default-dockercfg-4nrlm\"" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759035 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520b5181-4c77-49eb-9f90-0471620e7129-istio-podinfo\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759100 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-socket\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759191 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-envoy\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759262 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxwp\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-kube-api-access-nxxwp\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759297 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-certs\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759328 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-data\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759479 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520b5181-4c77-49eb-9f90-0471620e7129-istiod-ca-cert\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759529 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-credential-socket\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.759722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.759610 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-istio-token\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.906776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.869460 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520b5181-4c77-49eb-9f90-0471620e7129-istiod-ca-cert\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.921527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.921490 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-credential-socket\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.924579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.924540 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-istio-token\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.925440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.925411 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520b5181-4c77-49eb-9f90-0471620e7129-istio-podinfo\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.925648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.925617 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-socket\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.925905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.925888 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-envoy\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.926093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.926079 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxwp\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-kube-api-access-nxxwp\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.926218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.926205 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-certs\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.926354 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.926335 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-data\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.927157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.896640 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520b5181-4c77-49eb-9f90-0471620e7129-istiod-ca-cert\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.959107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.959070 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-certs\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.973936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.973895 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-credential-socket\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.976094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.976062 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5"] Nov 03 19:11:07.981720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.978014 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-socket\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:07.993772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:07.993735 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-data\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:08.126201 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:08.126171 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520b5181-4c77-49eb-9f90-0471620e7129-istio-podinfo\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:08.159539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:08.144253 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-istio-token\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:08.159539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:08.148180 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxwp\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-kube-api-access-nxxwp\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:08.159539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:08.149691 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-envoy\") pod \"e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:08.220127 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:08.220075 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:09.191308 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:11:09.191269 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520b5181_4c77_49eb_9f90_0471620e7129.slice/crio-f8e60e7bc04b5a7720d47769e0777b5747bbcead58c7b422c314333531e33be1 WatchSource:0}: Error finding container f8e60e7bc04b5a7720d47769e0777b5747bbcead58c7b422c314333531e33be1: Status 404 returned error can't find the container with id f8e60e7bc04b5a7720d47769e0777b5747bbcead58c7b422c314333531e33be1 Nov 03 19:11:09.192884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:09.192851 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5"] Nov 03 19:11:09.196102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:09.196064 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:09.196216 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:09.196158 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:09.196272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:09.196230 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:09.905865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:09.905776 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" event={"ID":"520b5181-4c77-49eb-9f90-0471620e7129","Type":"ContainerStarted","Data":"4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66"} Nov 03 19:11:09.905865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:09.905842 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" event={"ID":"520b5181-4c77-49eb-9f90-0471620e7129","Type":"ContainerStarted","Data":"f8e60e7bc04b5a7720d47769e0777b5747bbcead58c7b422c314333531e33be1"} Nov 03 19:11:10.083454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.083401 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" podStartSLOduration=3.083384551 podStartE2EDuration="3.083384551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:10.078849393 +0000 UTC m=+1999.561845298" watchObservedRunningTime="2026-11-03 19:11:10.083384551 +0000 UTC m=+1999.566380465" Nov 03 19:11:10.234725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.222193 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:10.243722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.243556 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:10.494459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.494369 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8"] Nov 03 19:11:10.502116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.502086 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ac14ff6-jtrn8"] Nov 03 19:11:10.759099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.759021 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d34bca1-6692-4dd0-9696-ca291a40836d" path="/var/lib/kubelet/pods/9d34bca1-6692-4dd0-9696-ca291a40836d/volumes" Nov 03 19:11:10.916177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.916136 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:10.917110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:10.917088 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:11:15.086730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:15.078915 4948 generic.go:356] "Generic (PLEG): container finished" podID="dce38743-6104-4500-8cf0-0806409dce6e" containerID="6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87" exitCode=0 Nov 03 19:11:15.095999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:15.079796 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" event={"ID":"dce38743-6104-4500-8cf0-0806409dce6e","Type":"ContainerDied","Data":"6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87"} Nov 03 19:11:15.096878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:15.096856 4948 scope.go:122] "RemoveContainer" containerID="6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87" Nov 03 19:11:15.121648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:15.121358 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:15.121648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:15.121481 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:15.121648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:15.121530 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:15.351011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:15.350924 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:11:16.166721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:16.166175 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" event={"ID":"dce38743-6104-4500-8cf0-0806409dce6e","Type":"ContainerStarted","Data":"951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173"} Nov 03 19:11:16.635657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:16.635626 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:11:17.048687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:17.048646 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 19:11:17.048923 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:17.048717 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 19:11:17.096755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:17.096713 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:11:17.879600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:17.879561 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 19:11:17.880229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:17.880197 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 19:11:18.735153 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:18.735095 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6"] Nov 03 19:11:18.735834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:18.735796 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" containerID="cri-o://126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12" gracePeriod=30 Nov 03 19:11:18.913740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:18.908405 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 19:11:18.913740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:18.908481 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 19:11:19.747892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:19.747858 4948 patch_prober.go:29] interesting pod/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:11:19.748144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:19.748121 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:11:19.896849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:19.896807 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 19:11:19.897132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:19.897101 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 19:11:20.878721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:20.877811 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 19:11:20.878721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:20.877874 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 19:11:21.881724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:21.881347 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 19:11:21.881724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:21.881423 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 19:11:22.097726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:22.096766 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b"] Nov 03 19:11:22.123308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:22.123271 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75130f0c-fcb4b"] Nov 03 19:11:22.787826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:22.787782 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f475bdd3-4f45-4fdc-a884-0580f9450c75" path="/var/lib/kubelet/pods/f475bdd3-4f45-4fdc-a884-0580f9450c75/volumes" Nov 03 19:11:22.891450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:22.891417 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:11:22.892739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:22.892676 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:11:22.893533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:22.893513 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:11:24.052224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.052194 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:11:24.140027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.139992 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-certs\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140041 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-envoy\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140134 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-credential-socket\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140212 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istiod-ca-cert\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140267 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-token\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-token\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140318 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-certs" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:24.140518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140358 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-data\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140403 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-podinfo\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140461 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-kube-api-access-9nnkn\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-kube-api-access-9nnkn\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140469 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-credential-socket" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:24.140763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140540 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-socket\") pod \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\" (UID: \"9a9a4423-52ad-4f7b-b92e-49d008755cbe\") " Nov 03 19:11:24.140763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.140567 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istiod-ca-cert" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:11:24.141094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.141068 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-socket" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:24.141323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.141305 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.141418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.141332 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.141418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.141349 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.141418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.141365 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.141594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.141433 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-data" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:24.142590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.142569 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-token" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:11:24.142789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.142769 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-podinfo" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:11:24.142864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.142815 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-envoy" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:24.142953 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.142933 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-kube-api-access-9nnkn" pod "9a9a4423-52ad-4f7b-b92e-49d008755cbe" (UID: "9a9a4423-52ad-4f7b-b92e-49d008755cbe"). InnerVolumeSpecName "kube-api-access-9nnkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:11:24.243321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.243274 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.243321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.243312 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.243544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.243327 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.243544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.243341 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9a4423-52ad-4f7b-b92e-49d008755cbe-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.243544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.243354 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nnkn\" (UniqueName: \"kubernetes.io/projected/9a9a4423-52ad-4f7b-b92e-49d008755cbe-kube-api-access-9nnkn\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:24.732107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.732067 4948 generic.go:356] "Generic (PLEG): container finished" podID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerID="126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12" exitCode=0 Nov 03 19:11:24.732285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.732118 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" Nov 03 19:11:24.732285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.732145 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" event={"ID":"9a9a4423-52ad-4f7b-b92e-49d008755cbe","Type":"ContainerDied","Data":"126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12"} Nov 03 19:11:24.732285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.732202 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6" event={"ID":"9a9a4423-52ad-4f7b-b92e-49d008755cbe","Type":"ContainerDied","Data":"9dcd2011835ef9a6feab33b9c3db03ccc192d3672ec26c4f657fb9bd3818f3aa"} Nov 03 19:11:24.732285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.732226 4948 scope.go:122] "RemoveContainer" containerID="126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12" Nov 03 19:11:24.745623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.745596 4948 scope.go:122] "RemoveContainer" containerID="08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d" Nov 03 19:11:24.760037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.760008 4948 scope.go:122] "RemoveContainer" containerID="126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12" Nov 03 19:11:24.760335 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:11:24.760306 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12\": container with ID starting with 126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12 not found: ID does not exist" containerID="126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12" Nov 03 19:11:24.760403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.760338 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12"} err="failed to get container status \"126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12\": rpc error: code = NotFound desc = could not find container \"126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12\": container with ID starting with 126ca5d7658ab2353919f634e827166deb2aa03d07dba6be2b356c4cf2004a12 not found: ID does not exist" Nov 03 19:11:24.760403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.760356 4948 scope.go:122] "RemoveContainer" containerID="08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d" Nov 03 19:11:24.760666 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:11:24.760647 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d\": container with ID starting with 08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d not found: ID does not exist" containerID="08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d" Nov 03 19:11:24.760770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.760671 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d"} err="failed to get container status \"08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d\": rpc error: code = NotFound desc = could not find container \"08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d\": container with ID starting with 08e399286c6e69f46ec903560b4f22f082ec330aa093965820914a9270eee52d not found: ID does not exist" Nov 03 19:11:24.905585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.905553 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6"] Nov 03 19:11:24.958333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:24.958296 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5ac14ff6-openshift-default-5859bb68c7-d4zd6"] Nov 03 19:11:26.760477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:26.760444 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" path="/var/lib/kubelet/pods/9a9a4423-52ad-4f7b-b92e-49d008755cbe/volumes" Nov 03 19:11:27.405978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.404427 4948 patch_prober.go:29] interesting pod/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": read tcp 10.132.0.2:43926->10.132.0.200:15021: read: connection reset by peer" start-of-body= Nov 03 19:11:27.405978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.404542 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": read tcp 10.132.0.2:43926->10.132.0.200:15021: read: connection reset by peer" Nov 03 19:11:27.768437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.768401 4948 generic.go:356] "Generic (PLEG): container finished" podID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerID="be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1" exitCode=0 Nov 03 19:11:27.768926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.768483 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" event={"ID":"3d57cd9d-8092-445f-8a1a-b8773a31479f","Type":"ContainerDied","Data":"be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1"} Nov 03 19:11:27.769128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.769110 4948 scope.go:122] "RemoveContainer" containerID="be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1" Nov 03 19:11:27.771099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.771068 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:27.771216 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.771139 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:27.771216 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:27.771173 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:11:28.897906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:28.897277 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" event={"ID":"3d57cd9d-8092-445f-8a1a-b8773a31479f","Type":"ContainerStarted","Data":"8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02"} Nov 03 19:11:29.112527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.073536 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs"] Nov 03 19:11:29.112527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.075062 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" Nov 03 19:11:29.112527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.075086 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" Nov 03 19:11:29.112527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.075142 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" Nov 03 19:11:29.112527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.075150 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" Nov 03 19:11:29.112527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.075318 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" Nov 03 19:11:29.174180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.162731 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.182998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.168321 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs"] Nov 03 19:11:29.231781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.230600 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4320e5-dockercfg-w22cs\"" Nov 03 19:11:29.231781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.230943 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4320e5\"" Nov 03 19:11:29.239388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.239362 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.239718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.239681 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw52b\" (UniqueName: \"kubernetes.io/projected/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-kube-api-access-mw52b\") pod \"payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.341725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.341662 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.342000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.341984 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw52b\" (UniqueName: \"kubernetes.io/projected/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-kube-api-access-mw52b\") pod \"payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.342412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.342389 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.403950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.403913 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw52b\" (UniqueName: \"kubernetes.io/projected/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-kube-api-access-mw52b\") pod \"payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.533167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.533134 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:29.744654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.744615 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:11:29.745597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.745565 4948 patch_prober.go:29] interesting pod/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 19:11:29.745770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:29.745612 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 19:11:30.011168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:30.008312 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs"] Nov 03 19:11:30.745300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:30.745213 4948 patch_prober.go:29] interesting pod/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 19:11:30.745495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:30.745301 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 19:11:30.931473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:30.931434 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" event={"ID":"78e977a1-c195-4a17-aecb-3bcb2a30f3ed","Type":"ContainerStarted","Data":"22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f"} Nov 03 19:11:30.931653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:30.931495 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" event={"ID":"78e977a1-c195-4a17-aecb-3bcb2a30f3ed","Type":"ContainerStarted","Data":"78c901c3e5d85bf819700aa85fd04f0e366caf11e5ab50b1b19c247ea6301ed1"} Nov 03 19:11:30.931653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:30.931555 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:31.071775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:31.068596 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" podStartSLOduration=3.068573584 podStartE2EDuration="3.068573584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:31.048961643 +0000 UTC m=+2020.531957558" watchObservedRunningTime="2026-11-03 19:11:31.068573584 +0000 UTC m=+2020.551569499" Nov 03 19:11:31.745181 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:31.745138 4948 patch_prober.go:29] interesting pod/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 19:11:31.745394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:31.745237 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 19:11:32.745430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:32.745394 4948 patch_prober.go:29] interesting pod/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 19:11:32.746107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:32.745453 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 19:11:32.806440 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:11:32.806398 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:11:33.745827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:33.745776 4948 patch_prober.go:29] interesting pod/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 19:11:33.746298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:33.745836 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 19:11:34.805479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:34.804851 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:11:34.806340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:34.806304 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:11:34.807187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:34.807160 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:11:40.794553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.794517 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv"] Nov 03 19:11:40.795507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.795488 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9a4423-52ad-4f7b-b92e-49d008755cbe" containerName="istio-proxy" Nov 03 19:11:40.799535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.799516 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:40.854126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.854094 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv"] Nov 03 19:11:40.895931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.895899 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-97e020-serving-cert\"" Nov 03 19:11:40.988689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.988654 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:40.988904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.988736 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a286ad3e-d341-42bb-a304-d92156e43b02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:40.988904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.988809 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkrl\" (UniqueName: \"kubernetes.io/projected/a286ad3e-d341-42bb-a304-d92156e43b02-kube-api-access-5tkrl\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:40.989010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:40.988989 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.092769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.091822 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.092769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.091894 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a286ad3e-d341-42bb-a304-d92156e43b02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.092769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.091987 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkrl\" (UniqueName: \"kubernetes.io/projected/a286ad3e-d341-42bb-a304-d92156e43b02-kube-api-access-5tkrl\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.092769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.092181 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.092769 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:11:41.092463 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-serving-cert: secret "maas-api-e2e-shared-b-w1-97e020-serving-cert" not found Nov 03 19:11:41.092769 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:11:41.092543 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls podName:a286ad3e-d341-42bb-a304-d92156e43b02 nodeName:}" failed. No retries permitted until 2026-11-03 19:11:41.592515256 +0000 UTC m=+2031.075511161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls") pod "maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" (UID: "a286ad3e-d341-42bb-a304-d92156e43b02") : secret "maas-api-e2e-shared-b-w1-97e020-serving-cert" not found Nov 03 19:11:41.105795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.105749 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a286ad3e-d341-42bb-a304-d92156e43b02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.108638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.108611 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.164977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.164946 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkrl\" (UniqueName: \"kubernetes.io/projected/a286ad3e-d341-42bb-a304-d92156e43b02-kube-api-access-5tkrl\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.604023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.603986 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.606675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.606648 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.714044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.714007 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:41.944352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:41.944318 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:11:42.109570 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:11:42.109531 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda286ad3e_d341_42bb_a304_d92156e43b02.slice/crio-8ffd5fd5c61b6d8aa7dbb6697bc11378f90ffdb494da326b21d7acd265a08f20 WatchSource:0}: Error finding container 8ffd5fd5c61b6d8aa7dbb6697bc11378f90ffdb494da326b21d7acd265a08f20: Status 404 returned error can't find the container with id 8ffd5fd5c61b6d8aa7dbb6697bc11378f90ffdb494da326b21d7acd265a08f20 Nov 03 19:11:42.129073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:42.129033 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv"] Nov 03 19:11:42.144243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:42.144202 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" event={"ID":"a286ad3e-d341-42bb-a304-d92156e43b02","Type":"ContainerStarted","Data":"8ffd5fd5c61b6d8aa7dbb6697bc11378f90ffdb494da326b21d7acd265a08f20"} Nov 03 19:11:43.158003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:43.157949 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" event={"ID":"a286ad3e-d341-42bb-a304-d92156e43b02","Type":"ContainerStarted","Data":"7add863121078f8e74ab586c8c334c04f4a52eb26e65a57c2a9be76e2a4abf2e"} Nov 03 19:11:43.158524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:43.158054 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:11:43.310126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:43.310022 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" podStartSLOduration=2.433640134 podStartE2EDuration="3.310006926s" totalImagesPullingTime="876.366792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:43.307882958 +0000 UTC m=+2032.790878873" watchObservedRunningTime="2026-11-03 19:11:43.310006926 +0000 UTC m=+2032.793002840" Nov 03 19:11:49.175526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:11:49.175488 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:12:02.797680 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:12:02.797644 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:12:13.617269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:13.617188 4948 generic.go:356] "Generic (PLEG): container finished" podID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerID="a45a71dc16ec3ddf522bc0d2f8d16982d7c7945f602eb456b4edd527273cc3f5" exitCode=0 Nov 03 19:12:13.617755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:13.617270 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" event={"ID":"5d0263c0-0903-4b42-94e5-2d65d5823087","Type":"ContainerDied","Data":"a45a71dc16ec3ddf522bc0d2f8d16982d7c7945f602eb456b4edd527273cc3f5"} Nov 03 19:12:13.617997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:13.617973 4948 scope.go:122] "RemoveContainer" containerID="a45a71dc16ec3ddf522bc0d2f8d16982d7c7945f602eb456b4edd527273cc3f5" Nov 03 19:12:13.620190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:13.620158 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:13.620303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:13.620246 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:13.620368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:13.620298 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:12:14.044279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:14.044245 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:12:14.633784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:14.633744 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" event={"ID":"5d0263c0-0903-4b42-94e5-2d65d5823087","Type":"ContainerStarted","Data":"73fb8443a70ed3117408bbb82babdda1bac22f02673ebd2ebcd0e0ffc197ea11"} Nov 03 19:12:15.533225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:15.533189 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:12:15.534445 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:15.534402 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:12:15.534576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:15.534467 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 19:12:16.533473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:16.533433 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:12:16.533942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:16.533505 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 19:12:17.533268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:17.533227 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:12:17.533496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:17.533290 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 19:12:18.533428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:18.533391 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:12:18.533612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:18.533458 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 19:12:19.534042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:19.534008 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:12:19.534546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:19.534066 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 19:12:20.538633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:20.538601 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:12:20.539146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:20.539131 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:12:20.539799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:20.539777 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:12:32.796849 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:12:32.796806 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:12:47.499972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.499934 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d"] Nov 03 19:12:47.504172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.504150 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.560815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.560777 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d"] Nov 03 19:12:47.565529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.565493 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-metrics-certs\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.565693 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.565548 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2lx\" (UniqueName: \"kubernetes.io/projected/920591b3-777c-434a-9c8e-d9cdd4dd2696-kube-api-access-nv2lx\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.565693 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.565591 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/920591b3-777c-434a-9c8e-d9cdd4dd2696-openshift-service-ca\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.565856 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.565716 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.605339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.605309 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-042f66-serving-cert\"" Nov 03 19:12:47.667356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.667322 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-metrics-certs\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.667578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.667362 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2lx\" (UniqueName: \"kubernetes.io/projected/920591b3-777c-434a-9c8e-d9cdd4dd2696-kube-api-access-nv2lx\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.667578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.667401 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/920591b3-777c-434a-9c8e-d9cdd4dd2696-openshift-service-ca\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.667578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.667512 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.667784 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:12:47.667752 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-serving-cert: secret "maas-api-e2e-api-a-042f66-serving-cert" not found Nov 03 19:12:47.667858 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:12:47.667844 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls podName:920591b3-777c-434a-9c8e-d9cdd4dd2696 nodeName:}" failed. No retries permitted until 2026-11-03 19:12:48.167820602 +0000 UTC m=+2097.650816507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls") pod "maas-api-e2e-api-a-042f66-5545f494f-wgr4d" (UID: "920591b3-777c-434a-9c8e-d9cdd4dd2696") : secret "maas-api-e2e-api-a-042f66-serving-cert" not found Nov 03 19:12:47.668725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.668684 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/920591b3-777c-434a-9c8e-d9cdd4dd2696-openshift-service-ca\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.671003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.670978 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-metrics-certs\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:47.760992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:47.739432 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2lx\" (UniqueName: \"kubernetes.io/projected/920591b3-777c-434a-9c8e-d9cdd4dd2696-kube-api-access-nv2lx\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:48.193958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:48.193909 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:48.196745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:48.196713 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls\") pod \"maas-api-e2e-api-a-042f66-5545f494f-wgr4d\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:48.421446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:48.421407 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:48.816105 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:12:48.816066 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920591b3_777c_434a_9c8e_d9cdd4dd2696.slice/crio-2e80b911c9704b1fab9e90540e56950c6f94de727d0f5d58277b0690d22ee458 WatchSource:0}: Error finding container 2e80b911c9704b1fab9e90540e56950c6f94de727d0f5d58277b0690d22ee458: Status 404 returned error can't find the container with id 2e80b911c9704b1fab9e90540e56950c6f94de727d0f5d58277b0690d22ee458 Nov 03 19:12:48.817064 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:48.817040 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d"] Nov 03 19:12:49.055959 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:49.055923 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" event={"ID":"920591b3-777c-434a-9c8e-d9cdd4dd2696","Type":"ContainerStarted","Data":"2e80b911c9704b1fab9e90540e56950c6f94de727d0f5d58277b0690d22ee458"} Nov 03 19:12:50.069026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:50.068925 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" event={"ID":"920591b3-777c-434a-9c8e-d9cdd4dd2696","Type":"ContainerStarted","Data":"b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a"} Nov 03 19:12:50.069026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:50.069017 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:50.193323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:50.193262 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" podStartSLOduration=2.312178016 podStartE2EDuration="3.193244403s" totalImagesPullingTime="881.066387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:50.188386208 +0000 UTC m=+2099.671382113" watchObservedRunningTime="2026-11-03 19:12:50.193244403 +0000 UTC m=+2099.676240316" Nov 03 19:12:55.510065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:55.510028 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:12:55.517784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:55.517762 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:12:55.783905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:55.783815 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:12:55.791902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:55.791878 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:12:56.084374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:56.084297 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:12:59.681198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:59.681164 4948 scope.go:122] "RemoveContainer" containerID="4c6a34d7cf35f3114919d59c06e53462a2039e7fd07f753be8999ade0a02b242" Nov 03 19:12:59.696873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:59.696847 4948 scope.go:122] "RemoveContainer" containerID="c337bed8b388b86fd6d777ffa2a9cd091e3ffc30a8b116b31588695e568e35b1" Nov 03 19:12:59.710016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:12:59.709994 4948 scope.go:122] "RemoveContainer" containerID="1fd3a75f1b367b821de9b7c6cc2277b34fc6097753077cd51aeb6ec5587ba339" Nov 03 19:13:02.042727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.042625 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6"] Nov 03 19:13:02.099526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.099490 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.112748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.104653 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6"] Nov 03 19:13:02.162488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.159853 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4320e5-serving-cert\"" Nov 03 19:13:02.192439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.192396 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01466848-e34f-4b3e-9789-9effd2374ef1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.192641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.192505 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msf9z\" (UniqueName: \"kubernetes.io/projected/01466848-e34f-4b3e-9789-9effd2374ef1-kube-api-access-msf9z\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.192641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.192593 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-metrics-certs\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.192819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.192797 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.308804 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.308764 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.309193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.309167 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01466848-e34f-4b3e-9789-9effd2374ef1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.309390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.309375 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msf9z\" (UniqueName: \"kubernetes.io/projected/01466848-e34f-4b3e-9789-9effd2374ef1-kube-api-access-msf9z\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.309566 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.309550 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-metrics-certs\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.316480 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:02.316184 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-serving-cert: secret "maas-api-e2e-inf-a-4320e5-serving-cert" not found Nov 03 19:13:02.316480 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:02.316263 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls podName:01466848-e34f-4b3e-9789-9effd2374ef1 nodeName:}" failed. No retries permitted until 2026-11-03 19:13:02.8162414 +0000 UTC m=+2112.299237305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls") pod "maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" (UID: "01466848-e34f-4b3e-9789-9effd2374ef1") : secret "maas-api-e2e-inf-a-4320e5-serving-cert" not found Nov 03 19:13:02.317618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.317553 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01466848-e34f-4b3e-9789-9effd2374ef1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.351366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.351332 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-metrics-certs\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.381765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.381585 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msf9z\" (UniqueName: \"kubernetes.io/projected/01466848-e34f-4b3e-9789-9effd2374ef1-kube-api-access-msf9z\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.798189 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:02.798146 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:13:02.820236 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.820197 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:02.822845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:02.822819 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls\") pod \"maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:03.021372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.021337 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:03.269277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.269242 4948 generic.go:356] "Generic (PLEG): container finished" podID="acc16153-2d17-4bb4-a11b-efb0355158be" containerID="f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce" exitCode=0 Nov 03 19:13:03.269806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.269316 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" event={"ID":"acc16153-2d17-4bb4-a11b-efb0355158be","Type":"ContainerDied","Data":"f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce"} Nov 03 19:13:03.269886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.269834 4948 scope.go:122] "RemoveContainer" containerID="f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce" Nov 03 19:13:03.271825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.271794 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:13:03.271938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.271864 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:13:03.271938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.271893 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:13:03.417344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:03.417311 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6"] Nov 03 19:13:03.420284 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:13:03.420257 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01466848_e34f_4b3e_9789_9effd2374ef1.slice/crio-4ebcc3a3a7ad8d6862e1980a55755e0edbb3823e0cbc815f0063dba327265190 WatchSource:0}: Error finding container 4ebcc3a3a7ad8d6862e1980a55755e0edbb3823e0cbc815f0063dba327265190: Status 404 returned error can't find the container with id 4ebcc3a3a7ad8d6862e1980a55755e0edbb3823e0cbc815f0063dba327265190 Nov 03 19:13:04.286817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:04.286780 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" event={"ID":"acc16153-2d17-4bb4-a11b-efb0355158be","Type":"ContainerStarted","Data":"717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c"} Nov 03 19:13:04.288657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:04.288625 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" event={"ID":"01466848-e34f-4b3e-9789-9effd2374ef1","Type":"ContainerStarted","Data":"4ebcc3a3a7ad8d6862e1980a55755e0edbb3823e0cbc815f0063dba327265190"} Nov 03 19:13:04.392369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:04.392329 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:13:04.393651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:04.393611 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 19:13:04.393811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:04.393662 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 19:13:05.310309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:05.310266 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" event={"ID":"01466848-e34f-4b3e-9789-9effd2374ef1","Type":"ContainerStarted","Data":"6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f"} Nov 03 19:13:05.310844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:05.310391 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:05.333731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:05.333668 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:13:05.392712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:05.392654 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 19:13:05.392930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:05.392736 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 19:13:05.443328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:05.443276 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" podStartSLOduration=3.620122008 podStartE2EDuration="4.44326263s" totalImagesPullingTime="823.140622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:05.437189137 +0000 UTC m=+2114.920185052" watchObservedRunningTime="2026-11-03 19:13:05.44326263 +0000 UTC m=+2114.926258544" Nov 03 19:13:06.393222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:06.393181 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 19:13:06.393684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:06.393251 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 19:13:07.392546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:07.392502 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 19:13:07.392797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:07.392592 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 19:13:08.071723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:08.069811 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg"] Nov 03 19:13:08.102765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:08.102725 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095d4a9f-bm7hg"] Nov 03 19:13:08.393335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:08.393246 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 19:13:08.393511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:08.393330 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 19:13:08.786973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:08.786932 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535fd0ff-312c-41d7-99f8-e04d5bafb029" path="/var/lib/kubelet/pods/535fd0ff-312c-41d7-99f8-e04d5bafb029/volumes" Nov 03 19:13:09.399775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:09.399712 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 19:13:09.400282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:09.399796 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 19:13:09.425521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:09.425486 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz"] Nov 03 19:13:09.425764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:09.425742 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" containerID="cri-o://3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa" gracePeriod=30 Nov 03 19:13:10.398338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:10.398301 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:13:10.399685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:10.399658 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:13:11.032844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:11.032809 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:13:11.033266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:11.032891 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:13:12.343604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:12.343571 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:13:14.753927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.753896 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 19:13:14.930072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.930034 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-token\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-token\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.930853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.930821 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-envoy\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.930913 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-kube-api-access-79kfs\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-kube-api-access-79kfs\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.930974 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-credential-socket\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.931049 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-certs\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.931141 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-data\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.931304 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-socket\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.931337 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-credential-socket" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:14.931468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.931459 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-podinfo\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.931488 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istiod-ca-cert\") pod \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\" (UID: \"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8\") " Nov 03 19:13:14.931689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.931661 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-socket" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:14.932965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.932097 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-data" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:14.932965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.932371 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:14.932965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.932395 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:14.932965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.932411 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:14.933324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.933241 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-token" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:14.933472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.933439 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-certs" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:14.934104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.933850 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istiod-ca-cert" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:14.934104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.933908 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-kube-api-access-79kfs" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "kube-api-access-79kfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:14.934498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.934473 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-envoy" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:14.934617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:14.934603 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-podinfo" pod "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" (UID: "2d3829a1-7c3f-40d9-8b16-3f0d302e55a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:15.034406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.034314 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:15.034406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.034347 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:15.034406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.034356 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:15.034406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.034368 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:15.034406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.034382 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kfs\" (UniqueName: \"kubernetes.io/projected/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-kube-api-access-79kfs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:15.034406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.034397 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:15.476041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.476009 4948 generic.go:356] "Generic (PLEG): container finished" podID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerID="3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa" exitCode=0 Nov 03 19:13:15.476211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.476065 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" Nov 03 19:13:15.476211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.476086 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" event={"ID":"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8","Type":"ContainerDied","Data":"3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa"} Nov 03 19:13:15.476211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.476140 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz" event={"ID":"2d3829a1-7c3f-40d9-8b16-3f0d302e55a8","Type":"ContainerDied","Data":"97b176e649b6bed327ab49002b57c374f9b409873b10f56108a462d82496dfcb"} Nov 03 19:13:15.476211 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.476163 4948 scope.go:122] "RemoveContainer" containerID="3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa" Nov 03 19:13:15.493931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.493912 4948 scope.go:122] "RemoveContainer" containerID="3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa" Nov 03 19:13:15.494223 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:15.494202 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa\": container with ID starting with 3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa not found: ID does not exist" containerID="3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa" Nov 03 19:13:15.494265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.494236 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa"} err="failed to get container status \"3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa\": rpc error: code = NotFound desc = could not find container \"3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa\": container with ID starting with 3c8ae5ae50fa2914559c70c105be6c772bd587ed7871efffb7c40cac59c6d5aa not found: ID does not exist" Nov 03 19:13:15.649849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.649816 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz"] Nov 03 19:13:15.700654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:15.700616 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-phbjz"] Nov 03 19:13:16.762329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:16.762285 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" path="/var/lib/kubelet/pods/2d3829a1-7c3f-40d9-8b16-3f0d302e55a8/volumes" Nov 03 19:13:28.090033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:28.089991 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg"] Nov 03 19:13:28.208490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:28.208451 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt"] Nov 03 19:13:32.799096 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:32.799047 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:13:33.090966 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.090843 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" podUID="25f5715e-baf7-4c7a-bf28-b9440cb42e94" containerName="payload-processing" containerID="cri-o://d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45" gracePeriod=25 Nov 03 19:13:33.208885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.208818 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" podUID="c8559c03-65d3-4644-a58e-b63ef1cb475e" containerName="payload-pre-processing" containerID="cri-o://935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4" gracePeriod=25 Nov 03 19:13:33.431525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.431498 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:13:33.533599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.533557 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f5715e-baf7-4c7a-bf28-b9440cb42e94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5715e-baf7-4c7a-bf28-b9440cb42e94-plugins-config-volume\") pod \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " Nov 03 19:13:33.533923 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.533902 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f5715e-baf7-4c7a-bf28-b9440cb42e94-kube-api-access-5vkxx\" (UniqueName: \"kubernetes.io/projected/25f5715e-baf7-4c7a-bf28-b9440cb42e94-kube-api-access-5vkxx\") pod \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\" (UID: \"25f5715e-baf7-4c7a-bf28-b9440cb42e94\") " Nov 03 19:13:33.534109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.534081 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f5715e-baf7-4c7a-bf28-b9440cb42e94-plugins-config-volume" pod "25f5715e-baf7-4c7a-bf28-b9440cb42e94" (UID: "25f5715e-baf7-4c7a-bf28-b9440cb42e94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:33.534589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.534565 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5715e-baf7-4c7a-bf28-b9440cb42e94-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:33.536686 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.536660 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f5715e-baf7-4c7a-bf28-b9440cb42e94-kube-api-access-5vkxx" pod "25f5715e-baf7-4c7a-bf28-b9440cb42e94" (UID: "25f5715e-baf7-4c7a-bf28-b9440cb42e94"). InnerVolumeSpecName "kube-api-access-5vkxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:33.568255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.568213 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:13:33.636029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.635940 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8559c03-65d3-4644-a58e-b63ef1cb475e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8559c03-65d3-4644-a58e-b63ef1cb475e-plugins-config-volume\") pod \"c8559c03-65d3-4644-a58e-b63ef1cb475e\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " Nov 03 19:13:33.636221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.636205 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8559c03-65d3-4644-a58e-b63ef1cb475e-kube-api-access-hgjrs\" (UniqueName: \"kubernetes.io/projected/c8559c03-65d3-4644-a58e-b63ef1cb475e-kube-api-access-hgjrs\") pod \"c8559c03-65d3-4644-a58e-b63ef1cb475e\" (UID: \"c8559c03-65d3-4644-a58e-b63ef1cb475e\") " Nov 03 19:13:33.636407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.636380 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8559c03-65d3-4644-a58e-b63ef1cb475e-plugins-config-volume" pod "c8559c03-65d3-4644-a58e-b63ef1cb475e" (UID: "c8559c03-65d3-4644-a58e-b63ef1cb475e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:33.636743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.636686 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8559c03-65d3-4644-a58e-b63ef1cb475e-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:33.636743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.636727 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vkxx\" (UniqueName: \"kubernetes.io/projected/25f5715e-baf7-4c7a-bf28-b9440cb42e94-kube-api-access-5vkxx\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:33.638332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.638312 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8559c03-65d3-4644-a58e-b63ef1cb475e-kube-api-access-hgjrs" pod "c8559c03-65d3-4644-a58e-b63ef1cb475e" (UID: "c8559c03-65d3-4644-a58e-b63ef1cb475e"). InnerVolumeSpecName "kube-api-access-hgjrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:33.699774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.699739 4948 generic.go:356] "Generic (PLEG): container finished" podID="c8559c03-65d3-4644-a58e-b63ef1cb475e" containerID="935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4" exitCode=0 Nov 03 19:13:33.699977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.699791 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" Nov 03 19:13:33.699977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.699791 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" event={"ID":"c8559c03-65d3-4644-a58e-b63ef1cb475e","Type":"ContainerDied","Data":"935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4"} Nov 03 19:13:33.699977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.699928 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt" event={"ID":"c8559c03-65d3-4644-a58e-b63ef1cb475e","Type":"ContainerDied","Data":"a234bae995797e515ea434566402bfe67a7534643ba8c319d6902be0f6617e99"} Nov 03 19:13:33.699977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.699960 4948 scope.go:122] "RemoveContainer" containerID="935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4" Nov 03 19:13:33.701826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.701789 4948 generic.go:356] "Generic (PLEG): container finished" podID="25f5715e-baf7-4c7a-bf28-b9440cb42e94" containerID="d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45" exitCode=0 Nov 03 19:13:33.702032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.701964 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" event={"ID":"25f5715e-baf7-4c7a-bf28-b9440cb42e94","Type":"ContainerDied","Data":"d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45"} Nov 03 19:13:33.702032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.702012 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" Nov 03 19:13:33.702217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.702019 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg" event={"ID":"25f5715e-baf7-4c7a-bf28-b9440cb42e94","Type":"ContainerDied","Data":"cc26bd0284e3eb066920bd06658aa88261357aaee0e19a4a8305e168a1d205b2"} Nov 03 19:13:33.722567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.722539 4948 scope.go:122] "RemoveContainer" containerID="935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4" Nov 03 19:13:33.723095 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:33.723061 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4\": container with ID starting with 935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4 not found: ID does not exist" containerID="935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4" Nov 03 19:13:33.723227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.723108 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4"} err="failed to get container status \"935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4\": rpc error: code = NotFound desc = could not find container \"935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4\": container with ID starting with 935f49cd40988382ea418d727817b9055877e499a88e792259339e0c82e2d0c4 not found: ID does not exist" Nov 03 19:13:33.723227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.723136 4948 scope.go:122] "RemoveContainer" containerID="d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45" Nov 03 19:13:33.738942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.738904 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjrs\" (UniqueName: \"kubernetes.io/projected/c8559c03-65d3-4644-a58e-b63ef1cb475e-kube-api-access-hgjrs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:33.739337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.739310 4948 scope.go:122] "RemoveContainer" containerID="d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45" Nov 03 19:13:33.739747 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:33.739720 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45\": container with ID starting with d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45 not found: ID does not exist" containerID="d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45" Nov 03 19:13:33.739841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.739760 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45"} err="failed to get container status \"d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45\": rpc error: code = NotFound desc = could not find container \"d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45\": container with ID starting with d94365560d5393608a967d4d0da8f368a09783abff865c55d9263e407eef6d45 not found: ID does not exist" Nov 03 19:13:33.871787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.871732 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg"] Nov 03 19:13:33.876891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.876857 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bf3e1c-75d744986c-jnxtg"] Nov 03 19:13:33.987524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.987485 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt"] Nov 03 19:13:33.996224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:33.996185 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bf3e1c-6bc6cfb875-rhwwt"] Nov 03 19:13:34.757866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:34.757831 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f5715e-baf7-4c7a-bf28-b9440cb42e94" path="/var/lib/kubelet/pods/25f5715e-baf7-4c7a-bf28-b9440cb42e94/volumes" Nov 03 19:13:34.758355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:34.758337 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8559c03-65d3-4644-a58e-b63ef1cb475e" path="/var/lib/kubelet/pods/c8559c03-65d3-4644-a58e-b63ef1cb475e/volumes" Nov 03 19:13:48.421781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:48.421727 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722"] Nov 03 19:13:48.534760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:48.534730 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9"] Nov 03 19:13:51.323606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:51.323569 4948 generic.go:356] "Generic (PLEG): container finished" podID="520b5181-4c77-49eb-9f90-0471620e7129" containerID="4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66" exitCode=0 Nov 03 19:13:51.324263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:51.323645 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" event={"ID":"520b5181-4c77-49eb-9f90-0471620e7129","Type":"ContainerDied","Data":"4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66"} Nov 03 19:13:51.324263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:51.324241 4948 scope.go:122] "RemoveContainer" containerID="4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66" Nov 03 19:13:51.326436 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:51.326406 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:13:51.326535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:51.326480 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:13:51.326535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:51.326512 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:13:52.337474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:52.337433 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" event={"ID":"520b5181-4c77-49eb-9f90-0471620e7129","Type":"ContainerStarted","Data":"255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f"} Nov 03 19:13:53.220590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.220552 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:13:53.221838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.221807 4948 patch_prober.go:29] interesting pod/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 19:13:53.222009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.221859 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 19:13:53.422290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.422253 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" podUID="4cedb0d2-901b-4225-840d-09fabaf00746" containerName="payload-processing" containerID="cri-o://352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38" gracePeriod=25 Nov 03 19:13:53.535355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.535254 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" podUID="78788824-4241-4aec-b4c9-c677616c51c9" containerName="payload-pre-processing" containerID="cri-o://1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec" gracePeriod=25 Nov 03 19:13:53.850556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.850528 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:13:53.947064 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.947035 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cedb0d2-901b-4225-840d-09fabaf00746-kube-api-access-xs5nx\" (UniqueName: \"kubernetes.io/projected/4cedb0d2-901b-4225-840d-09fabaf00746-kube-api-access-xs5nx\") pod \"4cedb0d2-901b-4225-840d-09fabaf00746\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " Nov 03 19:13:53.947232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.947192 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cedb0d2-901b-4225-840d-09fabaf00746-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cedb0d2-901b-4225-840d-09fabaf00746-plugins-config-volume\") pod \"4cedb0d2-901b-4225-840d-09fabaf00746\" (UID: \"4cedb0d2-901b-4225-840d-09fabaf00746\") " Nov 03 19:13:53.948065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.948028 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cedb0d2-901b-4225-840d-09fabaf00746-plugins-config-volume" pod "4cedb0d2-901b-4225-840d-09fabaf00746" (UID: "4cedb0d2-901b-4225-840d-09fabaf00746"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:53.968728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.964169 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cedb0d2-901b-4225-840d-09fabaf00746-kube-api-access-xs5nx" pod "4cedb0d2-901b-4225-840d-09fabaf00746" (UID: "4cedb0d2-901b-4225-840d-09fabaf00746"). InnerVolumeSpecName "kube-api-access-xs5nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:53.997714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:53.997677 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:13:54.048265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.048225 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78788824-4241-4aec-b4c9-c677616c51c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78788824-4241-4aec-b4c9-c677616c51c9-plugins-config-volume\") pod \"78788824-4241-4aec-b4c9-c677616c51c9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " Nov 03 19:13:54.048444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.048427 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78788824-4241-4aec-b4c9-c677616c51c9-kube-api-access-4mnvw\" (UniqueName: \"kubernetes.io/projected/78788824-4241-4aec-b4c9-c677616c51c9-kube-api-access-4mnvw\") pod \"78788824-4241-4aec-b4c9-c677616c51c9\" (UID: \"78788824-4241-4aec-b4c9-c677616c51c9\") " Nov 03 19:13:54.048735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.048688 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78788824-4241-4aec-b4c9-c677616c51c9-plugins-config-volume" pod "78788824-4241-4aec-b4c9-c677616c51c9" (UID: "78788824-4241-4aec-b4c9-c677616c51c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:54.049157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.049135 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5nx\" (UniqueName: \"kubernetes.io/projected/4cedb0d2-901b-4225-840d-09fabaf00746-kube-api-access-xs5nx\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:54.049253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.049161 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78788824-4241-4aec-b4c9-c677616c51c9-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:54.049253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.049183 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cedb0d2-901b-4225-840d-09fabaf00746-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:54.069775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.068914 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78788824-4241-4aec-b4c9-c677616c51c9-kube-api-access-4mnvw" pod "78788824-4241-4aec-b4c9-c677616c51c9" (UID: "78788824-4241-4aec-b4c9-c677616c51c9"). InnerVolumeSpecName "kube-api-access-4mnvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:54.150896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.150853 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mnvw\" (UniqueName: \"kubernetes.io/projected/78788824-4241-4aec-b4c9-c677616c51c9-kube-api-access-4mnvw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:54.221863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.221824 4948 patch_prober.go:29] interesting pod/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 19:13:54.222073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.221893 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 19:13:54.386255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.386171 4948 generic.go:356] "Generic (PLEG): container finished" podID="78788824-4241-4aec-b4c9-c677616c51c9" containerID="1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec" exitCode=0 Nov 03 19:13:54.386255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.386219 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" Nov 03 19:13:54.387177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.387156 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" event={"ID":"78788824-4241-4aec-b4c9-c677616c51c9","Type":"ContainerDied","Data":"1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec"} Nov 03 19:13:54.387251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.387221 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9" event={"ID":"78788824-4241-4aec-b4c9-c677616c51c9","Type":"ContainerDied","Data":"b2f6c9eadb2dca7a245bd079faf627bedcae2e12b6b6bacf9c7c254da6ba69d6"} Nov 03 19:13:54.387302 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.387261 4948 scope.go:122] "RemoveContainer" containerID="1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec" Nov 03 19:13:54.388114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.388095 4948 generic.go:356] "Generic (PLEG): container finished" podID="4cedb0d2-901b-4225-840d-09fabaf00746" containerID="352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38" exitCode=0 Nov 03 19:13:54.388175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.388154 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" event={"ID":"4cedb0d2-901b-4225-840d-09fabaf00746","Type":"ContainerDied","Data":"352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38"} Nov 03 19:13:54.388219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.388183 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" event={"ID":"4cedb0d2-901b-4225-840d-09fabaf00746","Type":"ContainerDied","Data":"210cd953a287102856975f27dd10fecb317caff31640cbc15f0fb9389526dfc5"} Nov 03 19:13:54.388219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.388188 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722" Nov 03 19:13:54.402031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.402012 4948 scope.go:122] "RemoveContainer" containerID="1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec" Nov 03 19:13:54.402291 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:54.402270 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec\": container with ID starting with 1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec not found: ID does not exist" containerID="1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec" Nov 03 19:13:54.402387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.402306 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec"} err="failed to get container status \"1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec\": rpc error: code = NotFound desc = could not find container \"1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec\": container with ID starting with 1541b15855031756bd32bb6b0fdffc8bc535c147517598bc3e2572bbf8d1f2ec not found: ID does not exist" Nov 03 19:13:54.402387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.402326 4948 scope.go:122] "RemoveContainer" containerID="352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38" Nov 03 19:13:54.413876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.413855 4948 scope.go:122] "RemoveContainer" containerID="352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38" Nov 03 19:13:54.414231 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:13:54.414205 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38\": container with ID starting with 352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38 not found: ID does not exist" containerID="352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38" Nov 03 19:13:54.414282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.414241 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38"} err="failed to get container status \"352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38\": rpc error: code = NotFound desc = could not find container \"352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38\": container with ID starting with 352d0a36165a03df1d895e389b2d52946fd8eecbfa38bc67cb8ed8846fc5ec38 not found: ID does not exist" Nov 03 19:13:54.583416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.583377 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9"] Nov 03 19:13:54.587405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.587378 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ecf40a-857b7d867f-zgpw9"] Nov 03 19:13:54.683421 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.683391 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722"] Nov 03 19:13:54.691383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.691348 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ecf40a-5b5dd7dcff-nr722"] Nov 03 19:13:54.784964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.784037 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cedb0d2-901b-4225-840d-09fabaf00746" path="/var/lib/kubelet/pods/4cedb0d2-901b-4225-840d-09fabaf00746/volumes" Nov 03 19:13:54.784964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:54.784843 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78788824-4241-4aec-b4c9-c677616c51c9" path="/var/lib/kubelet/pods/78788824-4241-4aec-b4c9-c677616c51c9/volumes" Nov 03 19:13:55.221410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:55.221375 4948 patch_prober.go:29] interesting pod/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 19:13:55.221636 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:55.221437 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 19:13:55.916817 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:55.916784 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:13:56.221716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:56.221674 4948 patch_prober.go:29] interesting pod/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 19:13:56.221916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:56.221767 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 19:13:57.233722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:57.226354 4948 patch_prober.go:29] interesting pod/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 19:13:57.233722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:57.226438 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 19:13:58.226358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:58.226323 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:13:58.227656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:13:58.227632 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:14:02.797061 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:02.797016 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:14:09.096824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:09.096785 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424"] Nov 03 19:14:09.210810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:09.210777 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs"] Nov 03 19:14:14.097474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.097421 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" podUID="8008ec73-e7ca-4700-995b-2978b89688b2" containerName="payload-processing" containerID="cri-o://9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f" gracePeriod=25 Nov 03 19:14:14.211448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.211402 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" podUID="4c89c35e-a409-4f07-a68d-31884163253b" containerName="payload-pre-processing" containerID="cri-o://3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61" gracePeriod=25 Nov 03 19:14:14.416348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.416314 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:14:14.449444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.449320 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8008ec73-e7ca-4700-995b-2978b89688b2-kube-api-access-jhlwt\" (UniqueName: \"kubernetes.io/projected/8008ec73-e7ca-4700-995b-2978b89688b2-kube-api-access-jhlwt\") pod \"8008ec73-e7ca-4700-995b-2978b89688b2\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " Nov 03 19:14:14.449730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.449455 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8008ec73-e7ca-4700-995b-2978b89688b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8008ec73-e7ca-4700-995b-2978b89688b2-plugins-config-volume\") pod \"8008ec73-e7ca-4700-995b-2978b89688b2\" (UID: \"8008ec73-e7ca-4700-995b-2978b89688b2\") " Nov 03 19:14:14.450440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.450399 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8008ec73-e7ca-4700-995b-2978b89688b2-plugins-config-volume" pod "8008ec73-e7ca-4700-995b-2978b89688b2" (UID: "8008ec73-e7ca-4700-995b-2978b89688b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:14.459078 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.459040 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8008ec73-e7ca-4700-995b-2978b89688b2-kube-api-access-jhlwt" pod "8008ec73-e7ca-4700-995b-2978b89688b2" (UID: "8008ec73-e7ca-4700-995b-2978b89688b2"). InnerVolumeSpecName "kube-api-access-jhlwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:14.517475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.517437 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:14:14.551453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.551417 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c89c35e-a409-4f07-a68d-31884163253b-kube-api-access-qqld7\" (UniqueName: \"kubernetes.io/projected/4c89c35e-a409-4f07-a68d-31884163253b-kube-api-access-qqld7\") pod \"4c89c35e-a409-4f07-a68d-31884163253b\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " Nov 03 19:14:14.551635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.551621 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c89c35e-a409-4f07-a68d-31884163253b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c89c35e-a409-4f07-a68d-31884163253b-plugins-config-volume\") pod \"4c89c35e-a409-4f07-a68d-31884163253b\" (UID: \"4c89c35e-a409-4f07-a68d-31884163253b\") " Nov 03 19:14:14.552143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.552096 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c89c35e-a409-4f07-a68d-31884163253b-plugins-config-volume" pod "4c89c35e-a409-4f07-a68d-31884163253b" (UID: "4c89c35e-a409-4f07-a68d-31884163253b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:14.552483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.552462 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhlwt\" (UniqueName: \"kubernetes.io/projected/8008ec73-e7ca-4700-995b-2978b89688b2-kube-api-access-jhlwt\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:14.552551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.552489 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8008ec73-e7ca-4700-995b-2978b89688b2-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:14.552551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.552503 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c89c35e-a409-4f07-a68d-31884163253b-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:14.553844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.553807 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c89c35e-a409-4f07-a68d-31884163253b-kube-api-access-qqld7" pod "4c89c35e-a409-4f07-a68d-31884163253b" (UID: "4c89c35e-a409-4f07-a68d-31884163253b"). InnerVolumeSpecName "kube-api-access-qqld7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:14.654602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.654227 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqld7\" (UniqueName: \"kubernetes.io/projected/4c89c35e-a409-4f07-a68d-31884163253b-kube-api-access-qqld7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:14.752139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.752106 4948 generic.go:356] "Generic (PLEG): container finished" podID="8008ec73-e7ca-4700-995b-2978b89688b2" containerID="9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f" exitCode=0 Nov 03 19:14:14.752334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.752190 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" Nov 03 19:14:14.753694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.753673 4948 generic.go:356] "Generic (PLEG): container finished" podID="4c89c35e-a409-4f07-a68d-31884163253b" containerID="3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61" exitCode=0 Nov 03 19:14:14.753845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.753826 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" Nov 03 19:14:14.757492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.757461 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" event={"ID":"8008ec73-e7ca-4700-995b-2978b89688b2","Type":"ContainerDied","Data":"9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f"} Nov 03 19:14:14.757644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.757515 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424" event={"ID":"8008ec73-e7ca-4700-995b-2978b89688b2","Type":"ContainerDied","Data":"22264636b0b63862b8013c3ab6c2fa8a35d463e3f52324b523e0dad2d5261833"} Nov 03 19:14:14.757644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.757539 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" event={"ID":"4c89c35e-a409-4f07-a68d-31884163253b","Type":"ContainerDied","Data":"3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61"} Nov 03 19:14:14.757644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.757564 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs" event={"ID":"4c89c35e-a409-4f07-a68d-31884163253b","Type":"ContainerDied","Data":"efb4ecfb3232d8c90bffa0746d37e6c8a6af3f79bbcfa100ba47488ace749b16"} Nov 03 19:14:14.757644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.757593 4948 scope.go:122] "RemoveContainer" containerID="9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f" Nov 03 19:14:14.773189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.773162 4948 scope.go:122] "RemoveContainer" containerID="9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f" Nov 03 19:14:14.773557 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:14.773530 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f\": container with ID starting with 9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f not found: ID does not exist" containerID="9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f" Nov 03 19:14:14.773616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.773570 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f"} err="failed to get container status \"9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f\": rpc error: code = NotFound desc = could not find container \"9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f\": container with ID starting with 9b2081362295439243b070a2a019eff432ff44d5dcb5355e1712fd9df8ba561f not found: ID does not exist" Nov 03 19:14:14.773616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.773598 4948 scope.go:122] "RemoveContainer" containerID="3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61" Nov 03 19:14:14.784766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.784745 4948 scope.go:122] "RemoveContainer" containerID="3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61" Nov 03 19:14:14.785058 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:14.785035 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61\": container with ID starting with 3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61 not found: ID does not exist" containerID="3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61" Nov 03 19:14:14.785109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.785068 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61"} err="failed to get container status \"3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61\": rpc error: code = NotFound desc = could not find container \"3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61\": container with ID starting with 3ded64d8e29ef889549dc60d4df78dc6679c554a1a6c810d0043caa232948e61 not found: ID does not exist" Nov 03 19:14:14.923425 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.923342 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424"] Nov 03 19:14:14.928105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:14.928074 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba5674-7b8cb7b449-mt424"] Nov 03 19:14:15.039067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:15.039036 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs"] Nov 03 19:14:15.044045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:15.044018 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba5674-64fb9d85b7-kq6fs"] Nov 03 19:14:16.757622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:16.757578 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c89c35e-a409-4f07-a68d-31884163253b" path="/var/lib/kubelet/pods/4c89c35e-a409-4f07-a68d-31884163253b/volumes" Nov 03 19:14:16.758463 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:16.758443 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8008ec73-e7ca-4700-995b-2978b89688b2" path="/var/lib/kubelet/pods/8008ec73-e7ca-4700-995b-2978b89688b2/volumes" Nov 03 19:14:28.487968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:28.487904 4948 patch_prober.go:29] interesting pod/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:14:28.488518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:28.488007 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:14:31.248977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:31.248944 4948 generic.go:356] "Generic (PLEG): container finished" podID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerID="c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762" exitCode=0 Nov 03 19:14:31.249438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:31.249026 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" event={"ID":"ee4f8681-850f-4ce3-baa4-368173ab3179","Type":"ContainerDied","Data":"c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762"} Nov 03 19:14:31.249547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:31.249520 4948 scope.go:122] "RemoveContainer" containerID="c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762" Nov 03 19:14:31.251552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:31.251519 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:14:31.251673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:31.251606 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:14:31.251673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:31.251661 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:14:32.289134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:32.289095 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" event={"ID":"ee4f8681-850f-4ce3-baa4-368173ab3179","Type":"ContainerStarted","Data":"d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d"} Nov 03 19:14:32.713327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:32.713295 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:14:32.713592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:32.713570 4948 patch_prober.go:29] interesting pod/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 19:14:32.713647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:32.713617 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 19:14:32.813018 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:32.812982 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:14:33.713004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:33.712961 4948 patch_prober.go:29] interesting pod/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 19:14:33.713564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:33.713016 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 19:14:34.716399 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:34.716354 4948 patch_prober.go:29] interesting pod/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 19:14:34.716888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:34.716437 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 19:14:35.592340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:35.592298 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d"] Nov 03 19:14:35.592932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:35.592872 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" podUID="920591b3-777c-434a-9c8e-d9cdd4dd2696" containerName="maas-api" containerID="cri-o://b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a" gracePeriod=30 Nov 03 19:14:35.713804 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:35.713769 4948 patch_prober.go:29] interesting pod/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 19:14:35.713935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:35.713832 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 19:14:36.036720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.036670 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs"] Nov 03 19:14:36.607022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.604142 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:14:36.607022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.604680 4948 generic.go:356] "Generic (PLEG): container finished" podID="920591b3-777c-434a-9c8e-d9cdd4dd2696" containerID="b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a" exitCode=0 Nov 03 19:14:36.607022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.604853 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" event={"ID":"920591b3-777c-434a-9c8e-d9cdd4dd2696","Type":"ContainerDied","Data":"b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a"} Nov 03 19:14:36.607022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.604928 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" event={"ID":"920591b3-777c-434a-9c8e-d9cdd4dd2696","Type":"ContainerDied","Data":"2e80b911c9704b1fab9e90540e56950c6f94de727d0f5d58277b0690d22ee458"} Nov 03 19:14:36.607022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.604969 4948 scope.go:122] "RemoveContainer" containerID="b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a" Nov 03 19:14:36.620526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.620503 4948 scope.go:122] "RemoveContainer" containerID="b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a" Nov 03 19:14:36.620868 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:36.620844 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a\": container with ID starting with b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a not found: ID does not exist" containerID="b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a" Nov 03 19:14:36.621002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.620879 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a"} err="failed to get container status \"b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a\": rpc error: code = NotFound desc = could not find container \"b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a\": container with ID starting with b3b0737b46662ed65c267d617f237e2bc8970024a911315699aa4abf7645217a not found: ID does not exist" Nov 03 19:14:36.688240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.688203 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls\") pod \"920591b3-777c-434a-9c8e-d9cdd4dd2696\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " Nov 03 19:14:36.688412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.688383 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920591b3-777c-434a-9c8e-d9cdd4dd2696-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/920591b3-777c-434a-9c8e-d9cdd4dd2696-openshift-service-ca\") pod \"920591b3-777c-434a-9c8e-d9cdd4dd2696\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " Nov 03 19:14:36.688461 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.688437 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920591b3-777c-434a-9c8e-d9cdd4dd2696-kube-api-access-nv2lx\" (UniqueName: \"kubernetes.io/projected/920591b3-777c-434a-9c8e-d9cdd4dd2696-kube-api-access-nv2lx\") pod \"920591b3-777c-434a-9c8e-d9cdd4dd2696\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " Nov 03 19:14:36.688593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.688576 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-metrics-certs\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-metrics-certs\") pod \"920591b3-777c-434a-9c8e-d9cdd4dd2696\" (UID: \"920591b3-777c-434a-9c8e-d9cdd4dd2696\") " Nov 03 19:14:36.689086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.689057 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920591b3-777c-434a-9c8e-d9cdd4dd2696-openshift-service-ca" pod "920591b3-777c-434a-9c8e-d9cdd4dd2696" (UID: "920591b3-777c-434a-9c8e-d9cdd4dd2696"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:36.689365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.689350 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/920591b3-777c-434a-9c8e-d9cdd4dd2696-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:36.691572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.691543 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls" pod "920591b3-777c-434a-9c8e-d9cdd4dd2696" (UID: "920591b3-777c-434a-9c8e-d9cdd4dd2696"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:36.691928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.691907 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-metrics-certs" pod "920591b3-777c-434a-9c8e-d9cdd4dd2696" (UID: "920591b3-777c-434a-9c8e-d9cdd4dd2696"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:36.692072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.692046 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920591b3-777c-434a-9c8e-d9cdd4dd2696-kube-api-access-nv2lx" pod "920591b3-777c-434a-9c8e-d9cdd4dd2696" (UID: "920591b3-777c-434a-9c8e-d9cdd4dd2696"). InnerVolumeSpecName "kube-api-access-nv2lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:36.715083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.714778 4948 patch_prober.go:29] interesting pod/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 19:14:36.715083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.714823 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 19:14:36.795061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.795021 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2lx\" (UniqueName: \"kubernetes.io/projected/920591b3-777c-434a-9c8e-d9cdd4dd2696-kube-api-access-nv2lx\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:36.795061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.795063 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:36.795332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:36.795080 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/920591b3-777c-434a-9c8e-d9cdd4dd2696-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:37.691720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:37.687591 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" Nov 03 19:14:37.734723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:37.725531 4948 patch_prober.go:29] interesting pod/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 19:14:37.734723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:37.725587 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 19:14:37.941777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:37.919335 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d"] Nov 03 19:14:38.006775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:38.004447 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d"] Nov 03 19:14:38.769330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:38.763115 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920591b3-777c-434a-9c8e-d9cdd4dd2696" path="/var/lib/kubelet/pods/920591b3-777c-434a-9c8e-d9cdd4dd2696/volumes" Nov 03 19:14:38.793685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:38.793650 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:14:38.794254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:38.794230 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:14:38.795177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:38.795157 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:14:39.093860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:39.093767 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-042f66-5545f494f-wgr4d" podUID="920591b3-777c-434a-9c8e-d9cdd4dd2696" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.219:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:14:41.038272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.038232 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" podUID="78e977a1-c195-4a17-aecb-3bcb2a30f3ed" containerName="payload-pre-processing" containerID="cri-o://22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f" gracePeriod=25 Nov 03 19:14:41.352942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.352909 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:14:41.470309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.470276 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-plugins-config-volume\") pod \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " Nov 03 19:14:41.470497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.470464 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-kube-api-access-mw52b\" (UniqueName: \"kubernetes.io/projected/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-kube-api-access-mw52b\") pod \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\" (UID: \"78e977a1-c195-4a17-aecb-3bcb2a30f3ed\") " Nov 03 19:14:41.473348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.473309 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-kube-api-access-mw52b" pod "78e977a1-c195-4a17-aecb-3bcb2a30f3ed" (UID: "78e977a1-c195-4a17-aecb-3bcb2a30f3ed"). InnerVolumeSpecName "kube-api-access-mw52b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:41.473608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.473590 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-plugins-config-volume" pod "78e977a1-c195-4a17-aecb-3bcb2a30f3ed" (UID: "78e977a1-c195-4a17-aecb-3bcb2a30f3ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:41.572225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.572136 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:41.572225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.572173 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw52b\" (UniqueName: \"kubernetes.io/projected/78e977a1-c195-4a17-aecb-3bcb2a30f3ed-kube-api-access-mw52b\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:41.709552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.709521 4948 generic.go:356] "Generic (PLEG): container finished" podID="78e977a1-c195-4a17-aecb-3bcb2a30f3ed" containerID="22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f" exitCode=0 Nov 03 19:14:41.709801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.709566 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" Nov 03 19:14:41.709801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.709608 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" event={"ID":"78e977a1-c195-4a17-aecb-3bcb2a30f3ed","Type":"ContainerDied","Data":"22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f"} Nov 03 19:14:41.709801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.709667 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs" event={"ID":"78e977a1-c195-4a17-aecb-3bcb2a30f3ed","Type":"ContainerDied","Data":"78c901c3e5d85bf819700aa85fd04f0e366caf11e5ab50b1b19c247ea6301ed1"} Nov 03 19:14:41.709801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.709710 4948 scope.go:122] "RemoveContainer" containerID="22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f" Nov 03 19:14:41.722747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.722687 4948 scope.go:122] "RemoveContainer" containerID="22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f" Nov 03 19:14:41.723092 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:41.723072 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f\": container with ID starting with 22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f not found: ID does not exist" containerID="22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f" Nov 03 19:14:41.723175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.723105 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f"} err="failed to get container status \"22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f\": rpc error: code = NotFound desc = could not find container \"22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f\": container with ID starting with 22b6dd46c8c466b77de36ddf42f624e4028d96879c036320d11a890e032bcc0f not found: ID does not exist" Nov 03 19:14:41.880406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.880323 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs"] Nov 03 19:14:41.930810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:41.930773 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4320e5-57dcbbd995-2cfzs"] Nov 03 19:14:42.756530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:42.756480 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e977a1-c195-4a17-aecb-3bcb2a30f3ed" path="/var/lib/kubelet/pods/78e977a1-c195-4a17-aecb-3bcb2a30f3ed/volumes" Nov 03 19:14:45.341719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:45.341673 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5"] Nov 03 19:14:45.342234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:45.341952 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" containerID="cri-o://255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f" gracePeriod=30 Nov 03 19:14:47.061666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.061625 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6"] Nov 03 19:14:47.062247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.061923 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" podUID="01466848-e34f-4b3e-9789-9effd2374ef1" containerName="maas-api" containerID="cri-o://6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f" gracePeriod=30 Nov 03 19:14:47.488725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.482173 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:14:47.601089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.600584 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-metrics-certs\") pod \"01466848-e34f-4b3e-9789-9effd2374ef1\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " Nov 03 19:14:47.601089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.600769 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls\") pod \"01466848-e34f-4b3e-9789-9effd2374ef1\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " Nov 03 19:14:47.601348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.601183 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01466848-e34f-4b3e-9789-9effd2374ef1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01466848-e34f-4b3e-9789-9effd2374ef1-openshift-service-ca\") pod \"01466848-e34f-4b3e-9789-9effd2374ef1\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " Nov 03 19:14:47.601348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.601314 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01466848-e34f-4b3e-9789-9effd2374ef1-kube-api-access-msf9z\" (UniqueName: \"kubernetes.io/projected/01466848-e34f-4b3e-9789-9effd2374ef1-kube-api-access-msf9z\") pod \"01466848-e34f-4b3e-9789-9effd2374ef1\" (UID: \"01466848-e34f-4b3e-9789-9effd2374ef1\") " Nov 03 19:14:47.602126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.602076 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01466848-e34f-4b3e-9789-9effd2374ef1-openshift-service-ca" pod "01466848-e34f-4b3e-9789-9effd2374ef1" (UID: "01466848-e34f-4b3e-9789-9effd2374ef1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:47.605538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.603935 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls" pod "01466848-e34f-4b3e-9789-9effd2374ef1" (UID: "01466848-e34f-4b3e-9789-9effd2374ef1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:47.610162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.608819 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-metrics-certs" pod "01466848-e34f-4b3e-9789-9effd2374ef1" (UID: "01466848-e34f-4b3e-9789-9effd2374ef1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:47.610162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.609068 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01466848-e34f-4b3e-9789-9effd2374ef1-kube-api-access-msf9z" pod "01466848-e34f-4b3e-9789-9effd2374ef1" (UID: "01466848-e34f-4b3e-9789-9effd2374ef1"). InnerVolumeSpecName "kube-api-access-msf9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:47.703126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.703093 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:47.703126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.703124 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01466848-e34f-4b3e-9789-9effd2374ef1-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:47.703339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.703138 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01466848-e34f-4b3e-9789-9effd2374ef1-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:47.703339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.703150 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msf9z\" (UniqueName: \"kubernetes.io/projected/01466848-e34f-4b3e-9789-9effd2374ef1-kube-api-access-msf9z\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:47.770815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.770776 4948 generic.go:356] "Generic (PLEG): container finished" podID="01466848-e34f-4b3e-9789-9effd2374ef1" containerID="6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f" exitCode=0 Nov 03 19:14:47.770815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.770813 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" Nov 03 19:14:47.771045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.770878 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" event={"ID":"01466848-e34f-4b3e-9789-9effd2374ef1","Type":"ContainerDied","Data":"6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f"} Nov 03 19:14:47.771045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.770930 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" event={"ID":"01466848-e34f-4b3e-9789-9effd2374ef1","Type":"ContainerDied","Data":"4ebcc3a3a7ad8d6862e1980a55755e0edbb3823e0cbc815f0063dba327265190"} Nov 03 19:14:47.771045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.770955 4948 scope.go:122] "RemoveContainer" containerID="6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f" Nov 03 19:14:47.783522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.783498 4948 scope.go:122] "RemoveContainer" containerID="6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f" Nov 03 19:14:47.783849 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:47.783826 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f\": container with ID starting with 6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f not found: ID does not exist" containerID="6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f" Nov 03 19:14:47.783921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.783859 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f"} err="failed to get container status \"6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f\": rpc error: code = NotFound desc = could not find container \"6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f\": container with ID starting with 6907b97d059f56d5f5af9a126e987fead22519438d3463d8861eee4fbdceaf9f not found: ID does not exist" Nov 03 19:14:47.941351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.941309 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6"] Nov 03 19:14:47.950371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:47.950339 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6"] Nov 03 19:14:48.055760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:48.055658 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf"] Nov 03 19:14:48.062100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:48.062069 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbf95cc-7z5sf"] Nov 03 19:14:48.829223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:48.829183 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01466848-e34f-4b3e-9789-9effd2374ef1" path="/var/lib/kubelet/pods/01466848-e34f-4b3e-9789-9effd2374ef1/volumes" Nov 03 19:14:48.829943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:48.829919 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937a0f9e-1110-490c-9034-3b5c67f8696a" path="/var/lib/kubelet/pods/937a0f9e-1110-490c-9034-3b5c67f8696a/volumes" Nov 03 19:14:50.354497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.354446 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4320e5-7867d759c-7pzk6" podUID="01466848-e34f-4b3e-9789-9effd2374ef1" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.220:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:14:50.677063 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.677035 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:14:50.801765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.801683 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-socket\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.801982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.801833 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-certs\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.801982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.801873 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-credential-socket\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.801982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.801948 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-kube-api-access-nxxwp\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-kube-api-access-nxxwp\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.802160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802016 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-envoy\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.802160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802065 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520b5181-4c77-49eb-9f90-0471620e7129-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520b5181-4c77-49eb-9f90-0471620e7129-istiod-ca-cert\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.802160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802141 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-data\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.802325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802162 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-istio-token\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-istio-token\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.802325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802009 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-socket" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:50.802325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802190 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-credential-socket" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:50.802325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802177 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-certs" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:50.802325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802212 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/520b5181-4c77-49eb-9f90-0471620e7129-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520b5181-4c77-49eb-9f90-0471620e7129-istio-podinfo\") pod \"520b5181-4c77-49eb-9f90-0471620e7129\" (UID: \"520b5181-4c77-49eb-9f90-0471620e7129\") " Nov 03 19:14:50.802600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802471 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-data" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:50.802832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802809 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520b5181-4c77-49eb-9f90-0471620e7129-istiod-ca-cert" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:50.802999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802960 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.802999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.802990 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.803192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.803005 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.803192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.803022 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520b5181-4c77-49eb-9f90-0471620e7129-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.803192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.803038 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.804805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.804772 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-kube-api-access-nxxwp" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "kube-api-access-nxxwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:50.804915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.804804 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-envoy" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:14:50.805198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.805155 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/520b5181-4c77-49eb-9f90-0471620e7129-istio-podinfo" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:14:50.805282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.805229 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-istio-token" pod "520b5181-4c77-49eb-9f90-0471620e7129" (UID: "520b5181-4c77-49eb-9f90-0471620e7129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:50.861911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.861872 4948 generic.go:356] "Generic (PLEG): container finished" podID="520b5181-4c77-49eb-9f90-0471620e7129" containerID="255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f" exitCode=0 Nov 03 19:14:50.862082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.861928 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" Nov 03 19:14:50.862082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.861952 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" event={"ID":"520b5181-4c77-49eb-9f90-0471620e7129","Type":"ContainerDied","Data":"255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f"} Nov 03 19:14:50.862082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.862024 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5" event={"ID":"520b5181-4c77-49eb-9f90-0471620e7129","Type":"ContainerDied","Data":"f8e60e7bc04b5a7720d47769e0777b5747bbcead58c7b422c314333531e33be1"} Nov 03 19:14:50.862082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.862053 4948 scope.go:122] "RemoveContainer" containerID="255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f" Nov 03 19:14:50.878111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.878085 4948 scope.go:122] "RemoveContainer" containerID="4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66" Nov 03 19:14:50.890987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.890959 4948 scope.go:122] "RemoveContainer" containerID="255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f" Nov 03 19:14:50.891361 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:50.891340 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f\": container with ID starting with 255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f not found: ID does not exist" containerID="255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f" Nov 03 19:14:50.891428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.891374 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f"} err="failed to get container status \"255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f\": rpc error: code = NotFound desc = could not find container \"255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f\": container with ID starting with 255843e2b75d32eb179e7bdbbef54979fc565a96a651918d03f2894c3c79ba0f not found: ID does not exist" Nov 03 19:14:50.891428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.891395 4948 scope.go:122] "RemoveContainer" containerID="4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66" Nov 03 19:14:50.891763 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:14:50.891736 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66\": container with ID starting with 4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66 not found: ID does not exist" containerID="4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66" Nov 03 19:14:50.891877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.891764 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66"} err="failed to get container status \"4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66\": rpc error: code = NotFound desc = could not find container \"4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66\": container with ID starting with 4b33591206c184a22c9efc475df4b4ab8b45e98217289216598190b0417edd66 not found: ID does not exist" Nov 03 19:14:50.904819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.904787 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxwp\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-kube-api-access-nxxwp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.907975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.905466 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520b5181-4c77-49eb-9f90-0471620e7129-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.907975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.905520 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520b5181-4c77-49eb-9f90-0471620e7129-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:50.907975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:50.905539 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520b5181-4c77-49eb-9f90-0471620e7129-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:51.068859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:51.068823 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5"] Nov 03 19:14:51.074462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:51.074427 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-042f66-openshift-default-7859c48d9c-lqvl5"] Nov 03 19:14:52.755952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:52.755918 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520b5181-4c77-49eb-9f90-0471620e7129" path="/var/lib/kubelet/pods/520b5181-4c77-49eb-9f90-0471620e7129/volumes" Nov 03 19:14:59.643541 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.643500 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx"] Nov 03 19:14:59.644306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644289 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" Nov 03 19:14:59.644306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644306 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644322 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c8559c03-65d3-4644-a58e-b63ef1cb475e" containerName="payload-pre-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644327 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8559c03-65d3-4644-a58e-b63ef1cb475e" containerName="payload-pre-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644338 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4c89c35e-a409-4f07-a68d-31884163253b" containerName="payload-pre-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644344 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c89c35e-a409-4f07-a68d-31884163253b" containerName="payload-pre-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644363 4948 cpu_manager.go:414] "containerMap: removing container" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644368 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644377 4948 cpu_manager.go:414] "containerMap: removing container" podUID="01466848-e34f-4b3e-9789-9effd2374ef1" containerName="maas-api" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644382 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01466848-e34f-4b3e-9789-9effd2374ef1" containerName="maas-api" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644400 4948 cpu_manager.go:414] "containerMap: removing container" podUID="78e977a1-c195-4a17-aecb-3bcb2a30f3ed" containerName="payload-pre-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644406 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e977a1-c195-4a17-aecb-3bcb2a30f3ed" containerName="payload-pre-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644416 4948 cpu_manager.go:414] "containerMap: removing container" podUID="25f5715e-baf7-4c7a-bf28-b9440cb42e94" containerName="payload-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644421 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f5715e-baf7-4c7a-bf28-b9440cb42e94" containerName="payload-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644427 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4cedb0d2-901b-4225-840d-09fabaf00746" containerName="payload-processing" Nov 03 19:14:59.644435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644432 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cedb0d2-901b-4225-840d-09fabaf00746" containerName="payload-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644447 4948 cpu_manager.go:414] "containerMap: removing container" podUID="920591b3-777c-434a-9c8e-d9cdd4dd2696" containerName="maas-api" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644452 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920591b3-777c-434a-9c8e-d9cdd4dd2696" containerName="maas-api" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644469 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8008ec73-e7ca-4700-995b-2978b89688b2" containerName="payload-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644473 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8008ec73-e7ca-4700-995b-2978b89688b2" containerName="payload-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644481 4948 cpu_manager.go:414] "containerMap: removing container" podUID="78788824-4241-4aec-b4c9-c677616c51c9" containerName="payload-pre-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644486 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78788824-4241-4aec-b4c9-c677616c51c9" containerName="payload-pre-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644593 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644602 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="01466848-e34f-4b3e-9789-9effd2374ef1" containerName="maas-api" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644614 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e977a1-c195-4a17-aecb-3bcb2a30f3ed" containerName="payload-pre-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644624 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f5715e-baf7-4c7a-bf28-b9440cb42e94" containerName="payload-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644634 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cedb0d2-901b-4225-840d-09fabaf00746" containerName="payload-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644641 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644649 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="920591b3-777c-434a-9c8e-d9cdd4dd2696" containerName="maas-api" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644657 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8008ec73-e7ca-4700-995b-2978b89688b2" containerName="payload-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644664 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="78788824-4241-4aec-b4c9-c677616c51c9" containerName="payload-pre-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644671 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3829a1-7c3f-40d9-8b16-3f0d302e55a8" containerName="istio-proxy" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644679 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8559c03-65d3-4644-a58e-b63ef1cb475e" containerName="payload-pre-processing" Nov 03 19:14:59.645228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.644687 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c89c35e-a409-4f07-a68d-31884163253b" containerName="payload-pre-processing" Nov 03 19:14:59.648667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.648645 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.701780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.701737 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx"] Nov 03 19:14:59.733842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.733801 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.734021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.733861 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvz7\" (UniqueName: \"kubernetes.io/projected/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-kube-api-access-vlvz7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.734021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.733966 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.747503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.747420 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:14:59.747672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.747518 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:14:59.835875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.835832 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.835875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.835878 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvz7\" (UniqueName: \"kubernetes.io/projected/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-kube-api-access-vlvz7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.836215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.835972 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.836620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.836594 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.838427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.838404 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.850308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.850281 4948 scope.go:122] "RemoveContainer" containerID="f70a8e89dc11e2174feb3fab577be7a133f05954cc6bcc3aea99a818f5bf0207" Nov 03 19:14:59.900428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.900399 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvz7\" (UniqueName: \"kubernetes.io/projected/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-kube-api-access-vlvz7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:14:59.967277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:14:59.967230 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:15:00.171505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.171393 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr"] Nov 03 19:15:00.172715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.172679 4948 cpu_manager.go:414] "containerMap: removing container" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" Nov 03 19:15:00.172851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.172737 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520b5181-4c77-49eb-9f90-0471620e7129" containerName="istio-proxy" Nov 03 19:15:00.176967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.176940 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r"] Nov 03 19:15:00.177117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.177072 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.181838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.181814 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.206049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.206016 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w"] Nov 03 19:15:00.210170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.210146 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.210990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.210967 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5"] Nov 03 19:15:00.228144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.228113 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.233539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.233511 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf"] Nov 03 19:15:00.238932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.238905 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.239084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.239053 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz"] Nov 03 19:15:00.241404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241368 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqbl\" (UniqueName: \"kubernetes.io/projected/961a3275-52b1-443b-adcc-43af86bddcef-kube-api-access-fnqbl\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.241547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241436 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9k7\" (UniqueName: \"kubernetes.io/projected/510004c0-3875-4da4-af50-af656c67668f-kube-api-access-8q9k7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.241547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241510 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb3eff7-234f-4b79-9e31-69e75efc0fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.241780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241564 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.241780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241614 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjhh\" (UniqueName: \"kubernetes.io/projected/2cb3eff7-234f-4b79-9e31-69e75efc0fce-kube-api-access-gkjhh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.241780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241644 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7qr\" (UniqueName: \"kubernetes.io/projected/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-kube-api-access-9b7qr\") pod \"maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.241780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241693 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961a3275-52b1-443b-adcc-43af86bddcef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.241780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241756 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtkc\" (UniqueName: \"kubernetes.io/projected/8cb756b1-a9d3-4235-980d-d9009a6226ac-kube-api-access-jqtkc\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.242045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241861 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510004c0-3875-4da4-af50-af656c67668f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.242045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.241918 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb756b1-a9d3-4235-980d-d9009a6226ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.244360 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.244339 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.256257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.256223 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f"] Nov 03 19:15:00.261237 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.261210 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:00.272103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.272071 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr"] Nov 03 19:15:00.280307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.280276 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r"] Nov 03 19:15:00.318199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.318154 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w"] Nov 03 19:15:00.325961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.325921 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5"] Nov 03 19:15:00.343388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343348 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510004c0-3875-4da4-af50-af656c67668f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.343601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343421 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb756b1-a9d3-4235-980d-d9009a6226ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.343601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343484 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27c8z\" (UniqueName: \"kubernetes.io/projected/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-kube-api-access-27c8z\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:00.343601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343555 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqbl\" (UniqueName: \"kubernetes.io/projected/961a3275-52b1-443b-adcc-43af86bddcef-kube-api-access-fnqbl\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.343601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343594 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb51c78-b827-447d-9603-8d18eb55fed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.343946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343618 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9k7\" (UniqueName: \"kubernetes.io/projected/510004c0-3875-4da4-af50-af656c67668f-kube-api-access-8q9k7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.343946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343688 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb3eff7-234f-4b79-9e31-69e75efc0fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.343946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343806 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.343946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343857 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjhh\" (UniqueName: \"kubernetes.io/projected/2cb3eff7-234f-4b79-9e31-69e75efc0fce-kube-api-access-gkjhh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.343946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343887 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7qr\" (UniqueName: \"kubernetes.io/projected/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-kube-api-access-9b7qr\") pod \"maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.343946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343915 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:00.343946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343947 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fgh\" (UniqueName: \"kubernetes.io/projected/afb51c78-b827-447d-9603-8d18eb55fed1-kube-api-access-m4fgh\") pod \"maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.344335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.343975 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961a3275-52b1-443b-adcc-43af86bddcef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.344335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.344015 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtkc\" (UniqueName: \"kubernetes.io/projected/8cb756b1-a9d3-4235-980d-d9009a6226ac-kube-api-access-jqtkc\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.345093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.345060 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510004c0-3875-4da4-af50-af656c67668f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.345531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.345507 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb3eff7-234f-4b79-9e31-69e75efc0fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.345733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.345710 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.345846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.345813 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961a3275-52b1-443b-adcc-43af86bddcef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.346711 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.346673 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb756b1-a9d3-4235-980d-d9009a6226ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.362455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.362266 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf"] Nov 03 19:15:00.378663 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:00.377679 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66babf83_5e76_4406_8c0b_a8a30ccbfe1c.slice/crio-66e8a008a9ccbce8626c3b15bbe7609869203c35bcbb45b313ff49f0c2516fb0 WatchSource:0}: Error finding container 66e8a008a9ccbce8626c3b15bbe7609869203c35bcbb45b313ff49f0c2516fb0: Status 404 returned error can't find the container with id 66e8a008a9ccbce8626c3b15bbe7609869203c35bcbb45b313ff49f0c2516fb0 Nov 03 19:15:00.379825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.379790 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz"] Nov 03 19:15:00.389795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.389763 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx"] Nov 03 19:15:00.412659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.412620 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtkc\" (UniqueName: \"kubernetes.io/projected/8cb756b1-a9d3-4235-980d-d9009a6226ac-kube-api-access-jqtkc\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.415145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.415114 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqbl\" (UniqueName: \"kubernetes.io/projected/961a3275-52b1-443b-adcc-43af86bddcef-kube-api-access-fnqbl\") pod \"maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.416866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.416832 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9k7\" (UniqueName: \"kubernetes.io/projected/510004c0-3875-4da4-af50-af656c67668f-kube-api-access-8q9k7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.417936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.417916 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjhh\" (UniqueName: \"kubernetes.io/projected/2cb3eff7-234f-4b79-9e31-69e75efc0fce-kube-api-access-gkjhh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.418414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.418396 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7qr\" (UniqueName: \"kubernetes.io/projected/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-kube-api-access-9b7qr\") pod \"maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.432349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.432310 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f"] Nov 03 19:15:00.445995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.445962 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27c8z\" (UniqueName: \"kubernetes.io/projected/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-kube-api-access-27c8z\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:00.446173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.446086 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb51c78-b827-447d-9603-8d18eb55fed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.446247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.446235 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:00.446307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.446278 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fgh\" (UniqueName: \"kubernetes.io/projected/afb51c78-b827-447d-9603-8d18eb55fed1-kube-api-access-m4fgh\") pod \"maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.447006 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.446986 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:00.447074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.447037 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb51c78-b827-447d-9603-8d18eb55fed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.495895 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.495856 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:00.505287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.505248 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:00.529912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.529871 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:00.546444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.546417 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:00.552552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.552529 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fgh\" (UniqueName: \"kubernetes.io/projected/afb51c78-b827-447d-9603-8d18eb55fed1-kube-api-access-m4fgh\") pod \"maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.552708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.552674 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27c8z\" (UniqueName: \"kubernetes.io/projected/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-kube-api-access-27c8z\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:00.563757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.563277 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:00.573888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.573573 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:00.581571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:00.581543 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:01.114202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.114159 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" event={"ID":"66babf83-5e76-4406-8c0b-a8a30ccbfe1c","Type":"ContainerStarted","Data":"0c3ce16e64f44f95529f0d53014b14cf724df586877f80b5a911fec5e12049c9"} Nov 03 19:15:01.114748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.114230 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" event={"ID":"66babf83-5e76-4406-8c0b-a8a30ccbfe1c","Type":"ContainerStarted","Data":"66e8a008a9ccbce8626c3b15bbe7609869203c35bcbb45b313ff49f0c2516fb0"} Nov 03 19:15:01.193441 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.193408 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w"] Nov 03 19:15:01.195493 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:01.195464 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961a3275_52b1_443b_adcc_43af86bddcef.slice/crio-44eb32847d9f4d83e1307cadf87dbb213168bc15833c2894dab66f1ce24b06ca WatchSource:0}: Error finding container 44eb32847d9f4d83e1307cadf87dbb213168bc15833c2894dab66f1ce24b06ca: Status 404 returned error can't find the container with id 44eb32847d9f4d83e1307cadf87dbb213168bc15833c2894dab66f1ce24b06ca Nov 03 19:15:01.199669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.199642 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r"] Nov 03 19:15:01.201300 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:01.201262 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb756b1_a9d3_4235_980d_d9009a6226ac.slice/crio-dae5b6686e07f9da4b49d6b2c5c47e0b6a10a110bf03842fc3eb84f47565f341 WatchSource:0}: Error finding container dae5b6686e07f9da4b49d6b2c5c47e0b6a10a110bf03842fc3eb84f47565f341: Status 404 returned error can't find the container with id dae5b6686e07f9da4b49d6b2c5c47e0b6a10a110bf03842fc3eb84f47565f341 Nov 03 19:15:01.204536 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:01.204510 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32555649_8e98_4fb1_a9a6_29b5a9d6b90e.slice/crio-fa37b97f677b5a8f38eb3c6a6636c64f6cc7114bce8c0a5aaecfce8204eed432 WatchSource:0}: Error finding container fa37b97f677b5a8f38eb3c6a6636c64f6cc7114bce8c0a5aaecfce8204eed432: Status 404 returned error can't find the container with id fa37b97f677b5a8f38eb3c6a6636c64f6cc7114bce8c0a5aaecfce8204eed432 Nov 03 19:15:01.206939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.206917 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f"] Nov 03 19:15:01.247142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.247003 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz"] Nov 03 19:15:01.253030 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:01.252998 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510004c0_3875_4da4_af50_af656c67668f.slice/crio-2ac06bc73953de93c0d360e3c336ad3357e7aa2f2d1ba841820c9132af711e6e WatchSource:0}: Error finding container 2ac06bc73953de93c0d360e3c336ad3357e7aa2f2d1ba841820c9132af711e6e: Status 404 returned error can't find the container with id 2ac06bc73953de93c0d360e3c336ad3357e7aa2f2d1ba841820c9132af711e6e Nov 03 19:15:01.253389 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:01.253368 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb3eff7_234f_4b79_9e31_69e75efc0fce.slice/crio-bebe2ba510cbeb559f02d1c496415c1c0859afc542a525b40c781f4059b196bd WatchSource:0}: Error finding container bebe2ba510cbeb559f02d1c496415c1c0859afc542a525b40c781f4059b196bd: Status 404 returned error can't find the container with id bebe2ba510cbeb559f02d1c496415c1c0859afc542a525b40c781f4059b196bd Nov 03 19:15:01.255523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.255494 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5"] Nov 03 19:15:01.271435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.271038 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr"] Nov 03 19:15:01.279160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.279132 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf"] Nov 03 19:15:01.280737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:01.280661 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" podStartSLOduration=2.280643067 podStartE2EDuration="2.280643067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:01.264313632 +0000 UTC m=+2230.747309550" watchObservedRunningTime="2026-11-03 19:15:01.280643067 +0000 UTC m=+2230.763638982" Nov 03 19:15:02.126693 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.126647 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" event={"ID":"961a3275-52b1-443b-adcc-43af86bddcef","Type":"ContainerStarted","Data":"24ff11fe290bf95df9dfa12ded823ae9e3cecbd150b5948d870f56aa6055fba1"} Nov 03 19:15:02.127203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.126797 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" event={"ID":"961a3275-52b1-443b-adcc-43af86bddcef","Type":"ContainerStarted","Data":"44eb32847d9f4d83e1307cadf87dbb213168bc15833c2894dab66f1ce24b06ca"} Nov 03 19:15:02.128582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.128557 4948 generic.go:356] "Generic (PLEG): container finished" podID="66babf83-5e76-4406-8c0b-a8a30ccbfe1c" containerID="0c3ce16e64f44f95529f0d53014b14cf724df586877f80b5a911fec5e12049c9" exitCode=0 Nov 03 19:15:02.128710 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.128623 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" event={"ID":"66babf83-5e76-4406-8c0b-a8a30ccbfe1c","Type":"ContainerDied","Data":"0c3ce16e64f44f95529f0d53014b14cf724df586877f80b5a911fec5e12049c9"} Nov 03 19:15:02.130350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.130324 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" event={"ID":"b77fb6c3-2ae1-4eed-8883-d39ddc4da728","Type":"ContainerStarted","Data":"70fb41b440d24062191a1bb9d498f1b743b1bc7d1f4f6b7e19dbf966f5920b5c"} Nov 03 19:15:02.130494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.130373 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" event={"ID":"b77fb6c3-2ae1-4eed-8883-d39ddc4da728","Type":"ContainerStarted","Data":"245a4613bf2b4983b78946d17e0384f12a67c50524fa93569d3f161b6f1fde45"} Nov 03 19:15:02.132019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.131996 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" event={"ID":"8cb756b1-a9d3-4235-980d-d9009a6226ac","Type":"ContainerStarted","Data":"0359dcd3c3108f8fe8e88a047f10994a270c47403224c8f5e267957e4b90f6eb"} Nov 03 19:15:02.132133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.132037 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" event={"ID":"8cb756b1-a9d3-4235-980d-d9009a6226ac","Type":"ContainerStarted","Data":"dae5b6686e07f9da4b49d6b2c5c47e0b6a10a110bf03842fc3eb84f47565f341"} Nov 03 19:15:02.133666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.133641 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerStarted","Data":"24845fa49530370815ab5cad418feb4a0befbf56a3225983abae15cec42bf608"} Nov 03 19:15:02.133783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.133687 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerStarted","Data":"bebe2ba510cbeb559f02d1c496415c1c0859afc542a525b40c781f4059b196bd"} Nov 03 19:15:02.135549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.135523 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" event={"ID":"afb51c78-b827-447d-9603-8d18eb55fed1","Type":"ContainerStarted","Data":"acd5d2589121c57ae2726fc9a61b5908539c9bdca435acd1b04e88b7e810eabd"} Nov 03 19:15:02.135649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.135574 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" event={"ID":"afb51c78-b827-447d-9603-8d18eb55fed1","Type":"ContainerStarted","Data":"c4e385274571fe2b6157c84f39371bdd99e9b080fdb68ce07d195c8459794afe"} Nov 03 19:15:02.137510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.137487 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerStarted","Data":"572f2edc70bea23454aebf839327e6128c8545917077c28ed9f7b8f25cfbd078"} Nov 03 19:15:02.137601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.137527 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerStarted","Data":"fa37b97f677b5a8f38eb3c6a6636c64f6cc7114bce8c0a5aaecfce8204eed432"} Nov 03 19:15:02.139767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.139747 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" event={"ID":"510004c0-3875-4da4-af50-af656c67668f","Type":"ContainerStarted","Data":"1f6c59c20e5236c00ba59e7dc249421d6d64300b68c397e2f66741e83fb834ef"} Nov 03 19:15:02.139879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.139795 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" event={"ID":"510004c0-3875-4da4-af50-af656c67668f","Type":"ContainerStarted","Data":"2ac06bc73953de93c0d360e3c336ad3357e7aa2f2d1ba841820c9132af711e6e"} Nov 03 19:15:02.246615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.246558 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" podStartSLOduration=2.24654459 podStartE2EDuration="2.24654459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.243210965 +0000 UTC m=+2231.726206882" watchObservedRunningTime="2026-11-03 19:15:02.24654459 +0000 UTC m=+2231.729540504" Nov 03 19:15:02.369580 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.369509 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" podStartSLOduration=2.369487618 podStartE2EDuration="2.369487618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.362122122 +0000 UTC m=+2231.845118046" watchObservedRunningTime="2026-11-03 19:15:02.369487618 +0000 UTC m=+2231.852483533" Nov 03 19:15:02.478576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.478515 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" podStartSLOduration=2.478499472 podStartE2EDuration="2.478499472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.477797302 +0000 UTC m=+2231.960793217" watchObservedRunningTime="2026-11-03 19:15:02.478499472 +0000 UTC m=+2231.961495385" Nov 03 19:15:02.599265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.598662 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" podStartSLOduration=2.598640272 podStartE2EDuration="2.598640272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.595328532 +0000 UTC m=+2232.078324445" watchObservedRunningTime="2026-11-03 19:15:02.598640272 +0000 UTC m=+2232.081636187" Nov 03 19:15:02.798048 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:02.798004 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:15:02.835303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.835242 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" podStartSLOduration=2.835228763 podStartE2EDuration="2.835228763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.833160654 +0000 UTC m=+2232.316156569" watchObservedRunningTime="2026-11-03 19:15:02.835228763 +0000 UTC m=+2232.318224712" Nov 03 19:15:02.835751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:02.835716 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" podStartSLOduration=2.835695164 podStartE2EDuration="2.835695164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.716905641 +0000 UTC m=+2232.199901582" watchObservedRunningTime="2026-11-03 19:15:02.835695164 +0000 UTC m=+2232.318691100" Nov 03 19:15:03.069471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.069400 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" podStartSLOduration=3.069378233 podStartE2EDuration="3.069378233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:03.067472559 +0000 UTC m=+2232.550468474" watchObservedRunningTime="2026-11-03 19:15:03.069378233 +0000 UTC m=+2232.552374148" Nov 03 19:15:03.152231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.152191 4948 generic.go:356] "Generic (PLEG): container finished" podID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerID="70fb41b440d24062191a1bb9d498f1b743b1bc7d1f4f6b7e19dbf966f5920b5c" exitCode=60 Nov 03 19:15:03.152683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.152280 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" event={"ID":"b77fb6c3-2ae1-4eed-8883-d39ddc4da728","Type":"ContainerDied","Data":"70fb41b440d24062191a1bb9d498f1b743b1bc7d1f4f6b7e19dbf966f5920b5c"} Nov 03 19:15:03.153008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.152989 4948 scope.go:122] "RemoveContainer" containerID="70fb41b440d24062191a1bb9d498f1b743b1bc7d1f4f6b7e19dbf966f5920b5c" Nov 03 19:15:03.154354 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.154326 4948 generic.go:356] "Generic (PLEG): container finished" podID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerID="0359dcd3c3108f8fe8e88a047f10994a270c47403224c8f5e267957e4b90f6eb" exitCode=60 Nov 03 19:15:03.154458 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.154420 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" event={"ID":"8cb756b1-a9d3-4235-980d-d9009a6226ac","Type":"ContainerDied","Data":"0359dcd3c3108f8fe8e88a047f10994a270c47403224c8f5e267957e4b90f6eb"} Nov 03 19:15:03.154685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.154670 4948 scope.go:122] "RemoveContainer" containerID="0359dcd3c3108f8fe8e88a047f10994a270c47403224c8f5e267957e4b90f6eb" Nov 03 19:15:03.158971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.158946 4948 generic.go:356] "Generic (PLEG): container finished" podID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerID="24845fa49530370815ab5cad418feb4a0befbf56a3225983abae15cec42bf608" exitCode=60 Nov 03 19:15:03.159089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.159037 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerDied","Data":"24845fa49530370815ab5cad418feb4a0befbf56a3225983abae15cec42bf608"} Nov 03 19:15:03.159315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.159283 4948 scope.go:122] "RemoveContainer" containerID="24845fa49530370815ab5cad418feb4a0befbf56a3225983abae15cec42bf608" Nov 03 19:15:03.161547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.161522 4948 generic.go:356] "Generic (PLEG): container finished" podID="afb51c78-b827-447d-9603-8d18eb55fed1" containerID="acd5d2589121c57ae2726fc9a61b5908539c9bdca435acd1b04e88b7e810eabd" exitCode=60 Nov 03 19:15:03.161639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.161613 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" event={"ID":"afb51c78-b827-447d-9603-8d18eb55fed1","Type":"ContainerDied","Data":"acd5d2589121c57ae2726fc9a61b5908539c9bdca435acd1b04e88b7e810eabd"} Nov 03 19:15:03.162091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.161904 4948 scope.go:122] "RemoveContainer" containerID="acd5d2589121c57ae2726fc9a61b5908539c9bdca435acd1b04e88b7e810eabd" Nov 03 19:15:03.163926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.163903 4948 generic.go:356] "Generic (PLEG): container finished" podID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerID="572f2edc70bea23454aebf839327e6128c8545917077c28ed9f7b8f25cfbd078" exitCode=60 Nov 03 19:15:03.164060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.164036 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerDied","Data":"572f2edc70bea23454aebf839327e6128c8545917077c28ed9f7b8f25cfbd078"} Nov 03 19:15:03.164187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.164173 4948 scope.go:122] "RemoveContainer" containerID="572f2edc70bea23454aebf839327e6128c8545917077c28ed9f7b8f25cfbd078" Nov 03 19:15:03.166346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.166320 4948 generic.go:356] "Generic (PLEG): container finished" podID="510004c0-3875-4da4-af50-af656c67668f" containerID="1f6c59c20e5236c00ba59e7dc249421d6d64300b68c397e2f66741e83fb834ef" exitCode=60 Nov 03 19:15:03.166449 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.166400 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" event={"ID":"510004c0-3875-4da4-af50-af656c67668f","Type":"ContainerDied","Data":"1f6c59c20e5236c00ba59e7dc249421d6d64300b68c397e2f66741e83fb834ef"} Nov 03 19:15:03.166647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.166629 4948 scope.go:122] "RemoveContainer" containerID="1f6c59c20e5236c00ba59e7dc249421d6d64300b68c397e2f66741e83fb834ef" Nov 03 19:15:03.174260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.173415 4948 generic.go:356] "Generic (PLEG): container finished" podID="961a3275-52b1-443b-adcc-43af86bddcef" containerID="24ff11fe290bf95df9dfa12ded823ae9e3cecbd150b5948d870f56aa6055fba1" exitCode=60 Nov 03 19:15:03.174260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.174025 4948 scope.go:122] "RemoveContainer" containerID="24ff11fe290bf95df9dfa12ded823ae9e3cecbd150b5948d870f56aa6055fba1" Nov 03 19:15:03.174755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.174466 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" event={"ID":"961a3275-52b1-443b-adcc-43af86bddcef","Type":"ContainerDied","Data":"24ff11fe290bf95df9dfa12ded823ae9e3cecbd150b5948d870f56aa6055fba1"} Nov 03 19:15:03.507648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.507616 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:15:03.610027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.609940 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-openshift-ca\") pod \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " Nov 03 19:15:03.610203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.610086 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-xks-ca\") pod \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " Nov 03 19:15:03.610257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.610241 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-kube-api-access-vlvz7\" (UniqueName: \"kubernetes.io/projected/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-kube-api-access-vlvz7\") pod \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\" (UID: \"66babf83-5e76-4406-8c0b-a8a30ccbfe1c\") " Nov 03 19:15:03.610329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.610303 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-openshift-ca" pod "66babf83-5e76-4406-8c0b-a8a30ccbfe1c" (UID: "66babf83-5e76-4406-8c0b-a8a30ccbfe1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:03.610864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.610833 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:03.612434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.612412 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-xks-ca" pod "66babf83-5e76-4406-8c0b-a8a30ccbfe1c" (UID: "66babf83-5e76-4406-8c0b-a8a30ccbfe1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:03.612805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.612785 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-kube-api-access-vlvz7" pod "66babf83-5e76-4406-8c0b-a8a30ccbfe1c" (UID: "66babf83-5e76-4406-8c0b-a8a30ccbfe1c"). InnerVolumeSpecName "kube-api-access-vlvz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:03.712069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.712030 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:03.712069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.712063 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvz7\" (UniqueName: \"kubernetes.io/projected/66babf83-5e76-4406-8c0b-a8a30ccbfe1c-kube-api-access-vlvz7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:03.728689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.728642 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf"] Nov 03 19:15:03.744678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:03.744622 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv"] Nov 03 19:15:04.186414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.186382 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" event={"ID":"961a3275-52b1-443b-adcc-43af86bddcef","Type":"ContainerStarted","Data":"fe6230b0031d1b1d150979ab316dba8dff3db4e2c0b5d0ee0c88176976974c75"} Nov 03 19:15:04.188398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.188372 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" Nov 03 19:15:04.188549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.188372 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx" event={"ID":"66babf83-5e76-4406-8c0b-a8a30ccbfe1c","Type":"ContainerDied","Data":"66e8a008a9ccbce8626c3b15bbe7609869203c35bcbb45b313ff49f0c2516fb0"} Nov 03 19:15:04.188549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.188501 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e8a008a9ccbce8626c3b15bbe7609869203c35bcbb45b313ff49f0c2516fb0" Nov 03 19:15:04.190761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.190736 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" event={"ID":"b77fb6c3-2ae1-4eed-8883-d39ddc4da728","Type":"ContainerStarted","Data":"dc5fd943007ca7c8f9ddd94dc34716c99c74b7710006be79cbce8f9c3042b9ab"} Nov 03 19:15:04.192965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.192911 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" event={"ID":"8cb756b1-a9d3-4235-980d-d9009a6226ac","Type":"ContainerStarted","Data":"4efd6fb431995a354ff04dbbab513c0753d5f35f12c01ebb1a05afd84fae3328"} Nov 03 19:15:04.195011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.194985 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerStarted","Data":"d2d9e4bae2067d1c2257358f6e32f559aacf6aebabb45bbca552a29e988f9422"} Nov 03 19:15:04.197241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.197220 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" event={"ID":"afb51c78-b827-447d-9603-8d18eb55fed1","Type":"ContainerStarted","Data":"155cccc6b65b82488d71bec3e056f2edd82443c452c88940d86f63ea947263ae"} Nov 03 19:15:04.199754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.199729 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerStarted","Data":"c5ec49311fb440cc96fe423fd4fb3ecf680b2d904bffc77c1fcd7ebfbc4440b4"} Nov 03 19:15:04.201940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:04.201918 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" event={"ID":"510004c0-3875-4da4-af50-af656c67668f","Type":"ContainerStarted","Data":"f5333eadbcefc340c3e7b4221b10c73d85dbec76a58044bf2c23ce8ac1c246bb"} Nov 03 19:15:05.213220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.213183 4948 generic.go:356] "Generic (PLEG): container finished" podID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerID="d2d9e4bae2067d1c2257358f6e32f559aacf6aebabb45bbca552a29e988f9422" exitCode=60 Nov 03 19:15:05.213775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.213266 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerDied","Data":"d2d9e4bae2067d1c2257358f6e32f559aacf6aebabb45bbca552a29e988f9422"} Nov 03 19:15:05.213775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.213340 4948 scope.go:122] "RemoveContainer" containerID="24845fa49530370815ab5cad418feb4a0befbf56a3225983abae15cec42bf608" Nov 03 19:15:05.213775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.213723 4948 scope.go:122] "RemoveContainer" containerID="d2d9e4bae2067d1c2257358f6e32f559aacf6aebabb45bbca552a29e988f9422" Nov 03 19:15:05.214180 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:05.214157 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr_odh-ai-gateway-infra(2cb3eff7-234f-4b79-9e31-69e75efc0fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" Nov 03 19:15:05.215396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.215377 4948 generic.go:356] "Generic (PLEG): container finished" podID="afb51c78-b827-447d-9603-8d18eb55fed1" containerID="155cccc6b65b82488d71bec3e056f2edd82443c452c88940d86f63ea947263ae" exitCode=60 Nov 03 19:15:05.215490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.215466 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" event={"ID":"afb51c78-b827-447d-9603-8d18eb55fed1","Type":"ContainerDied","Data":"155cccc6b65b82488d71bec3e056f2edd82443c452c88940d86f63ea947263ae"} Nov 03 19:15:05.215742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.215725 4948 scope.go:122] "RemoveContainer" containerID="155cccc6b65b82488d71bec3e056f2edd82443c452c88940d86f63ea947263ae" Nov 03 19:15:05.216097 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:05.216074 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz_odh-ai-gateway-infra(afb51c78-b827-447d-9603-8d18eb55fed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" Nov 03 19:15:05.217471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.217449 4948 generic.go:356] "Generic (PLEG): container finished" podID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerID="c5ec49311fb440cc96fe423fd4fb3ecf680b2d904bffc77c1fcd7ebfbc4440b4" exitCode=60 Nov 03 19:15:05.217597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.217517 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerDied","Data":"c5ec49311fb440cc96fe423fd4fb3ecf680b2d904bffc77c1fcd7ebfbc4440b4"} Nov 03 19:15:05.217853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.217834 4948 scope.go:122] "RemoveContainer" containerID="c5ec49311fb440cc96fe423fd4fb3ecf680b2d904bffc77c1fcd7ebfbc4440b4" Nov 03 19:15:05.218171 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:05.218151 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f_odh-ai-gateway-infra(32555649-8e98-4fb1-a9a6-29b5a9d6b90e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" Nov 03 19:15:05.219581 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.219559 4948 generic.go:356] "Generic (PLEG): container finished" podID="510004c0-3875-4da4-af50-af656c67668f" containerID="f5333eadbcefc340c3e7b4221b10c73d85dbec76a58044bf2c23ce8ac1c246bb" exitCode=60 Nov 03 19:15:05.219671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.219635 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" event={"ID":"510004c0-3875-4da4-af50-af656c67668f","Type":"ContainerDied","Data":"f5333eadbcefc340c3e7b4221b10c73d85dbec76a58044bf2c23ce8ac1c246bb"} Nov 03 19:15:05.219887 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.219870 4948 scope.go:122] "RemoveContainer" containerID="f5333eadbcefc340c3e7b4221b10c73d85dbec76a58044bf2c23ce8ac1c246bb" Nov 03 19:15:05.220137 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:05.220118 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5_odh-ai-gateway-infra(510004c0-3875-4da4-af50-af656c67668f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" podUID="510004c0-3875-4da4-af50-af656c67668f" Nov 03 19:15:05.221863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.221842 4948 generic.go:356] "Generic (PLEG): container finished" podID="961a3275-52b1-443b-adcc-43af86bddcef" containerID="fe6230b0031d1b1d150979ab316dba8dff3db4e2c0b5d0ee0c88176976974c75" exitCode=60 Nov 03 19:15:05.221952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.221868 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" event={"ID":"961a3275-52b1-443b-adcc-43af86bddcef","Type":"ContainerDied","Data":"fe6230b0031d1b1d150979ab316dba8dff3db4e2c0b5d0ee0c88176976974c75"} Nov 03 19:15:05.222132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.222118 4948 scope.go:122] "RemoveContainer" containerID="fe6230b0031d1b1d150979ab316dba8dff3db4e2c0b5d0ee0c88176976974c75" Nov 03 19:15:05.222368 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:05.222351 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w_odh-ai-gateway-infra(961a3275-52b1-443b-adcc-43af86bddcef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" podUID="961a3275-52b1-443b-adcc-43af86bddcef" Nov 03 19:15:05.224000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.223978 4948 generic.go:356] "Generic (PLEG): container finished" podID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerID="dc5fd943007ca7c8f9ddd94dc34716c99c74b7710006be79cbce8f9c3042b9ab" exitCode=60 Nov 03 19:15:05.224102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.224044 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" event={"ID":"b77fb6c3-2ae1-4eed-8883-d39ddc4da728","Type":"ContainerDied","Data":"dc5fd943007ca7c8f9ddd94dc34716c99c74b7710006be79cbce8f9c3042b9ab"} Nov 03 19:15:05.224262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.224248 4948 scope.go:122] "RemoveContainer" containerID="dc5fd943007ca7c8f9ddd94dc34716c99c74b7710006be79cbce8f9c3042b9ab" Nov 03 19:15:05.224489 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:05.224469 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf_odh-ai-gateway-infra(b77fb6c3-2ae1-4eed-8883-d39ddc4da728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" Nov 03 19:15:05.226939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.226920 4948 generic.go:356] "Generic (PLEG): container finished" podID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerID="4efd6fb431995a354ff04dbbab513c0753d5f35f12c01ebb1a05afd84fae3328" exitCode=60 Nov 03 19:15:05.227046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.226951 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" event={"ID":"8cb756b1-a9d3-4235-980d-d9009a6226ac","Type":"ContainerDied","Data":"4efd6fb431995a354ff04dbbab513c0753d5f35f12c01ebb1a05afd84fae3328"} Nov 03 19:15:05.227227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.227210 4948 scope.go:122] "RemoveContainer" containerID="4efd6fb431995a354ff04dbbab513c0753d5f35f12c01ebb1a05afd84fae3328" Nov 03 19:15:05.227465 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:05.227445 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r_odh-ai-gateway-infra(8cb756b1-a9d3-4235-980d-d9009a6226ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" Nov 03 19:15:05.228779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.228762 4948 scope.go:122] "RemoveContainer" containerID="acd5d2589121c57ae2726fc9a61b5908539c9bdca435acd1b04e88b7e810eabd" Nov 03 19:15:05.244589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.244565 4948 scope.go:122] "RemoveContainer" containerID="572f2edc70bea23454aebf839327e6128c8545917077c28ed9f7b8f25cfbd078" Nov 03 19:15:05.259102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.259079 4948 scope.go:122] "RemoveContainer" containerID="1f6c59c20e5236c00ba59e7dc249421d6d64300b68c397e2f66741e83fb834ef" Nov 03 19:15:05.273522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.273496 4948 scope.go:122] "RemoveContainer" containerID="24ff11fe290bf95df9dfa12ded823ae9e3cecbd150b5948d870f56aa6055fba1" Nov 03 19:15:05.288010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.287983 4948 scope.go:122] "RemoveContainer" containerID="70fb41b440d24062191a1bb9d498f1b743b1bc7d1f4f6b7e19dbf966f5920b5c" Nov 03 19:15:05.303717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:05.303677 4948 scope.go:122] "RemoveContainer" containerID="0359dcd3c3108f8fe8e88a047f10994a270c47403224c8f5e267957e4b90f6eb" Nov 03 19:15:06.239810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:06.239782 4948 scope.go:122] "RemoveContainer" containerID="fe6230b0031d1b1d150979ab316dba8dff3db4e2c0b5d0ee0c88176976974c75" Nov 03 19:15:06.240247 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:06.240179 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w_odh-ai-gateway-infra(961a3275-52b1-443b-adcc-43af86bddcef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" podUID="961a3275-52b1-443b-adcc-43af86bddcef" Nov 03 19:15:06.241801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:06.241778 4948 scope.go:122] "RemoveContainer" containerID="dc5fd943007ca7c8f9ddd94dc34716c99c74b7710006be79cbce8f9c3042b9ab" Nov 03 19:15:06.242121 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:06.242099 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf_odh-ai-gateway-infra(b77fb6c3-2ae1-4eed-8883-d39ddc4da728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" Nov 03 19:15:06.243472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:06.243452 4948 scope.go:122] "RemoveContainer" containerID="4efd6fb431995a354ff04dbbab513c0753d5f35f12c01ebb1a05afd84fae3328" Nov 03 19:15:06.243787 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:06.243760 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r_odh-ai-gateway-infra(8cb756b1-a9d3-4235-980d-d9009a6226ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" Nov 03 19:15:06.245999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:06.245981 4948 scope.go:122] "RemoveContainer" containerID="d2d9e4bae2067d1c2257358f6e32f559aacf6aebabb45bbca552a29e988f9422" Nov 03 19:15:06.246280 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:06.246260 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr_odh-ai-gateway-infra(2cb3eff7-234f-4b79-9e31-69e75efc0fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" Nov 03 19:15:06.247651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:06.247633 4948 scope.go:122] "RemoveContainer" containerID="155cccc6b65b82488d71bec3e056f2edd82443c452c88940d86f63ea947263ae" Nov 03 19:15:06.247949 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:06.247928 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz_odh-ai-gateway-infra(afb51c78-b827-447d-9603-8d18eb55fed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" Nov 03 19:15:06.249279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:06.249262 4948 scope.go:122] "RemoveContainer" containerID="c5ec49311fb440cc96fe423fd4fb3ecf680b2d904bffc77c1fcd7ebfbc4440b4" Nov 03 19:15:06.249521 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:06.249498 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f_odh-ai-gateway-infra(32555649-8e98-4fb1-a9a6-29b5a9d6b90e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" Nov 03 19:15:06.250877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:06.250860 4948 scope.go:122] "RemoveContainer" containerID="f5333eadbcefc340c3e7b4221b10c73d85dbec76a58044bf2c23ce8ac1c246bb" Nov 03 19:15:06.251068 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:06.251052 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5_odh-ai-gateway-infra(510004c0-3875-4da4-af50-af656c67668f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" podUID="510004c0-3875-4da4-af50-af656c67668f" Nov 03 19:15:13.466533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.466490 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk"] Nov 03 19:15:13.467434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.467414 4948 cpu_manager.go:414] "containerMap: removing container" podUID="66babf83-5e76-4406-8c0b-a8a30ccbfe1c" containerName="revoke-keys" Nov 03 19:15:13.467482 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.467435 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66babf83-5e76-4406-8c0b-a8a30ccbfe1c" containerName="revoke-keys" Nov 03 19:15:13.467587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.467578 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="66babf83-5e76-4406-8c0b-a8a30ccbfe1c" containerName="revoke-keys" Nov 03 19:15:13.471217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.471197 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.525947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.525910 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk"] Nov 03 19:15:13.569907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.569878 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:15:13.628205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.628155 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.628422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.628395 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmqb\" (UniqueName: \"kubernetes.io/projected/7b909622-2a7b-48de-b2cc-50bc64b60f57-kube-api-access-tlmqb\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.628479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.628432 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.730028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.729938 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.730197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.730104 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmqb\" (UniqueName: \"kubernetes.io/projected/7b909622-2a7b-48de-b2cc-50bc64b60f57-kube-api-access-tlmqb\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.730197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.730135 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.730864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.730839 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.732440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.732417 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:13.792485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:13.792450 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmqb\" (UniqueName: \"kubernetes.io/projected/7b909622-2a7b-48de-b2cc-50bc64b60f57-kube-api-access-tlmqb\") pod \"maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:14.085118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.085025 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:14.382381 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.382295 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch"] Nov 03 19:15:14.382568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.382537 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" podUID="e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" containerName="maas-api" containerID="cri-o://7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599" gracePeriod=30 Nov 03 19:15:14.500546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.500514 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk"] Nov 03 19:15:14.502247 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:14.502223 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b909622_2a7b_48de_b2cc_50bc64b60f57.slice/crio-4a2ffad1e1ab593006fa55a83d6292a7630d124954f5ff9d36b446f43bb8383c WatchSource:0}: Error finding container 4a2ffad1e1ab593006fa55a83d6292a7630d124954f5ff9d36b446f43bb8383c: Status 404 returned error can't find the container with id 4a2ffad1e1ab593006fa55a83d6292a7630d124954f5ff9d36b446f43bb8383c Nov 03 19:15:14.732913 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.732863 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc"] Nov 03 19:15:14.755907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.747665 4948 scope.go:122] "RemoveContainer" containerID="4efd6fb431995a354ff04dbbab513c0753d5f35f12c01ebb1a05afd84fae3328" Nov 03 19:15:14.847587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.847553 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8"] Nov 03 19:15:14.851519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.851489 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:15:14.855451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.855420 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-kube-api-access-9mw5x\" (UniqueName: \"kubernetes.io/projected/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-kube-api-access-9mw5x\") pod \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " Nov 03 19:15:14.855653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.855474 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-metrics-certs\") pod \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " Nov 03 19:15:14.855808 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.855784 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-openshift-service-ca\") pod \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " Nov 03 19:15:14.855919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.855816 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls\") pod \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\" (UID: \"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698\") " Nov 03 19:15:14.856291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.856267 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-openshift-service-ca" pod "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" (UID: "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:14.856551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.856533 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:14.858464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.858439 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-metrics-certs" pod "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" (UID: "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:14.858568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.858481 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-kube-api-access-9mw5x" pod "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" (UID: "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698"). InnerVolumeSpecName "kube-api-access-9mw5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:14.858974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:14.858948 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls" pod "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" (UID: "e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:15.039782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.017641 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:15.040070 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.040031 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:15.040220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.040203 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mw5x\" (UniqueName: \"kubernetes.io/projected/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698-kube-api-access-9mw5x\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:15.481739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.479835 4948 generic.go:356] "Generic (PLEG): container finished" podID="e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" containerID="7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599" exitCode=0 Nov 03 19:15:15.481739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.480020 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" event={"ID":"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698","Type":"ContainerDied","Data":"7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599"} Nov 03 19:15:15.481739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.480076 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" event={"ID":"e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698","Type":"ContainerDied","Data":"217ecddceeacd97a90ac037855f2a8c277f6fbcaa4ce3f980db4b901e22bc479"} Nov 03 19:15:15.481739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.480105 4948 scope.go:122] "RemoveContainer" containerID="7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599" Nov 03 19:15:15.481739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.480282 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch" Nov 03 19:15:15.529057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.528155 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" event={"ID":"7b909622-2a7b-48de-b2cc-50bc64b60f57","Type":"ContainerStarted","Data":"b27588a904c574cfe3a383759f42eac5bad9b0cf41afb9def311c5c8227fecdf"} Nov 03 19:15:15.529057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.528216 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" event={"ID":"7b909622-2a7b-48de-b2cc-50bc64b60f57","Type":"ContainerStarted","Data":"4a2ffad1e1ab593006fa55a83d6292a7630d124954f5ff9d36b446f43bb8383c"} Nov 03 19:15:15.584412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.584387 4948 scope.go:122] "RemoveContainer" containerID="7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599" Nov 03 19:15:15.584770 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:15.584744 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599\": container with ID starting with 7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599 not found: ID does not exist" containerID="7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599" Nov 03 19:15:15.584824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.584783 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599"} err="failed to get container status \"7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599\": rpc error: code = NotFound desc = could not find container \"7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599\": container with ID starting with 7652db7cd1b02f47d461b5f1638962b433d0bc0806f5c99328e1715b3312c599 not found: ID does not exist" Nov 03 19:15:15.698636 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.698580 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" podStartSLOduration=2.698564252 podStartE2EDuration="2.698564252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:15.695850662 +0000 UTC m=+2245.178846575" watchObservedRunningTime="2026-11-03 19:15:15.698564252 +0000 UTC m=+2245.181560165" Nov 03 19:15:15.758504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.758468 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5"] Nov 03 19:15:15.759732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.759712 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" containerName="maas-api" Nov 03 19:15:15.759815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.759735 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" containerName="maas-api" Nov 03 19:15:15.759978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.759965 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" containerName="maas-api" Nov 03 19:15:15.763411 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.763391 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.815881 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.815833 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch"] Nov 03 19:15:15.822394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.822359 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5"] Nov 03 19:15:15.830868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.830820 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecf40a-5d69f4b87c-m27ch"] Nov 03 19:15:15.883766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.883725 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.884011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.883781 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.884364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.884328 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4hd\" (UniqueName: \"kubernetes.io/projected/c9f061e8-d50d-4099-b988-3cc62387009d-kube-api-access-ps4hd\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.986183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.986062 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4hd\" (UniqueName: \"kubernetes.io/projected/c9f061e8-d50d-4099-b988-3cc62387009d-kube-api-access-ps4hd\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.986397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.986376 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.986468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.986428 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.987623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.987595 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:15.990574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:15.990549 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:16.050039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.050002 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4hd\" (UniqueName: \"kubernetes.io/projected/c9f061e8-d50d-4099-b988-3cc62387009d-kube-api-access-ps4hd\") pod \"maas-api-revoke-sub-0568ad91-rpdn5\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:16.079200 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.079154 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:16.550732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.550674 4948 generic.go:356] "Generic (PLEG): container finished" podID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerID="304b98639c605b2f92bd0190cfc0dab15932e18170ad9a938e27b68a3fd5924e" exitCode=60 Nov 03 19:15:16.551614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.551586 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" event={"ID":"8cb756b1-a9d3-4235-980d-d9009a6226ac","Type":"ContainerDied","Data":"304b98639c605b2f92bd0190cfc0dab15932e18170ad9a938e27b68a3fd5924e"} Nov 03 19:15:16.551888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.551872 4948 scope.go:122] "RemoveContainer" containerID="4efd6fb431995a354ff04dbbab513c0753d5f35f12c01ebb1a05afd84fae3328" Nov 03 19:15:16.552539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.552521 4948 scope.go:122] "RemoveContainer" containerID="304b98639c605b2f92bd0190cfc0dab15932e18170ad9a938e27b68a3fd5924e" Nov 03 19:15:16.553109 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:16.553086 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r_odh-ai-gateway-infra(8cb756b1-a9d3-4235-980d-d9009a6226ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" Nov 03 19:15:16.636919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.626243 4948 generic.go:356] "Generic (PLEG): container finished" podID="7b909622-2a7b-48de-b2cc-50bc64b60f57" containerID="b27588a904c574cfe3a383759f42eac5bad9b0cf41afb9def311c5c8227fecdf" exitCode=0 Nov 03 19:15:16.636919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.626990 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" event={"ID":"7b909622-2a7b-48de-b2cc-50bc64b60f57","Type":"ContainerDied","Data":"b27588a904c574cfe3a383759f42eac5bad9b0cf41afb9def311c5c8227fecdf"} Nov 03 19:15:16.677746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.671969 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5"] Nov 03 19:15:16.835582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.829267 4948 scope.go:122] "RemoveContainer" containerID="d2d9e4bae2067d1c2257358f6e32f559aacf6aebabb45bbca552a29e988f9422" Nov 03 19:15:16.835582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.830492 4948 scope.go:122] "RemoveContainer" containerID="c5ec49311fb440cc96fe423fd4fb3ecf680b2d904bffc77c1fcd7ebfbc4440b4" Nov 03 19:15:16.898062 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:16.898024 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698" path="/var/lib/kubelet/pods/e8a3feeb-2e84-4e4e-b944-4fcdd5c5f698/volumes" Nov 03 19:15:17.633745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.633711 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f"] Nov 03 19:15:17.638801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.638773 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerStarted","Data":"98983e2595f29104dfcc677015638f70c82dea8ea661a7ebd344a8ca8170d123"} Nov 03 19:15:17.642228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.642198 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" event={"ID":"c9f061e8-d50d-4099-b988-3cc62387009d","Type":"ContainerStarted","Data":"125c7d122a2dc7722297558f452a3d5c1a0dd1555df12adda14780d086a1e238"} Nov 03 19:15:17.642390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.642245 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" event={"ID":"c9f061e8-d50d-4099-b988-3cc62387009d","Type":"ContainerStarted","Data":"24f9544c16d7cdbbd4fc2b5573a8deb3b78e0edf265c225a2cb189e0457e1669"} Nov 03 19:15:17.644396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.644371 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerStarted","Data":"b394f23eb01685994f3e864793af5b7f7cfec30968aba65fdcbfc099859d8ccf"} Nov 03 19:15:17.725517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.725478 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r"] Nov 03 19:15:17.799733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.798755 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2"] Nov 03 19:15:17.906582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:17.906551 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:18.016476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.016425 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" podStartSLOduration=3.016408841 podStartE2EDuration="3.016408841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:18.013817307 +0000 UTC m=+2247.496813227" watchObservedRunningTime="2026-11-03 19:15:18.016408841 +0000 UTC m=+2247.499404798" Nov 03 19:15:18.027290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.027263 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:18.100789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.092236 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-openshift-ca\") pod \"7b909622-2a7b-48de-b2cc-50bc64b60f57\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " Nov 03 19:15:18.100789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.092438 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b909622-2a7b-48de-b2cc-50bc64b60f57-kube-api-access-tlmqb\" (UniqueName: \"kubernetes.io/projected/7b909622-2a7b-48de-b2cc-50bc64b60f57-kube-api-access-tlmqb\") pod \"7b909622-2a7b-48de-b2cc-50bc64b60f57\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " Nov 03 19:15:18.100789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.092487 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-xks-ca\") pod \"7b909622-2a7b-48de-b2cc-50bc64b60f57\" (UID: \"7b909622-2a7b-48de-b2cc-50bc64b60f57\") " Nov 03 19:15:18.126042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.126001 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-openshift-ca" pod "7b909622-2a7b-48de-b2cc-50bc64b60f57" (UID: "7b909622-2a7b-48de-b2cc-50bc64b60f57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:18.134530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.134481 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b909622-2a7b-48de-b2cc-50bc64b60f57-kube-api-access-tlmqb" pod "7b909622-2a7b-48de-b2cc-50bc64b60f57" (UID: "7b909622-2a7b-48de-b2cc-50bc64b60f57"). InnerVolumeSpecName "kube-api-access-tlmqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:18.152332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.152272 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-xks-ca" pod "7b909622-2a7b-48de-b2cc-50bc64b60f57" (UID: "7b909622-2a7b-48de-b2cc-50bc64b60f57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:18.223231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.223195 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb756b1-a9d3-4235-980d-d9009a6226ac-kube-api-access-jqtkc\" (UniqueName: \"kubernetes.io/projected/8cb756b1-a9d3-4235-980d-d9009a6226ac-kube-api-access-jqtkc\") pod \"8cb756b1-a9d3-4235-980d-d9009a6226ac\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " Nov 03 19:15:18.223544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.223526 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb756b1-a9d3-4235-980d-d9009a6226ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb756b1-a9d3-4235-980d-d9009a6226ac-maas-api-service-ca\") pod \"8cb756b1-a9d3-4235-980d-d9009a6226ac\" (UID: \"8cb756b1-a9d3-4235-980d-d9009a6226ac\") " Nov 03 19:15:18.224631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.224588 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.230793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.230760 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmqb\" (UniqueName: \"kubernetes.io/projected/7b909622-2a7b-48de-b2cc-50bc64b60f57-kube-api-access-tlmqb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.235169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.235138 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb756b1-a9d3-4235-980d-d9009a6226ac-maas-api-service-ca" pod "8cb756b1-a9d3-4235-980d-d9009a6226ac" (UID: "8cb756b1-a9d3-4235-980d-d9009a6226ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:18.247324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.247235 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b909622-2a7b-48de-b2cc-50bc64b60f57-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.308232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.308077 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb756b1-a9d3-4235-980d-d9009a6226ac-kube-api-access-jqtkc" pod "8cb756b1-a9d3-4235-980d-d9009a6226ac" (UID: "8cb756b1-a9d3-4235-980d-d9009a6226ac"). InnerVolumeSpecName "kube-api-access-jqtkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:18.348404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.348362 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqtkc\" (UniqueName: \"kubernetes.io/projected/8cb756b1-a9d3-4235-980d-d9009a6226ac-kube-api-access-jqtkc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.348404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.348396 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb756b1-a9d3-4235-980d-d9009a6226ac-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.655230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.655146 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" Nov 03 19:15:18.655230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.655155 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r" event={"ID":"8cb756b1-a9d3-4235-980d-d9009a6226ac","Type":"ContainerDied","Data":"dae5b6686e07f9da4b49d6b2c5c47e0b6a10a110bf03842fc3eb84f47565f341"} Nov 03 19:15:18.655230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.655223 4948 scope.go:122] "RemoveContainer" containerID="304b98639c605b2f92bd0190cfc0dab15932e18170ad9a938e27b68a3fd5924e" Nov 03 19:15:18.657618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.657589 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" event={"ID":"7b909622-2a7b-48de-b2cc-50bc64b60f57","Type":"ContainerDied","Data":"4a2ffad1e1ab593006fa55a83d6292a7630d124954f5ff9d36b446f43bb8383c"} Nov 03 19:15:18.657767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.657644 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2ffad1e1ab593006fa55a83d6292a7630d124954f5ff9d36b446f43bb8383c" Nov 03 19:15:18.657847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.657772 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk" Nov 03 19:15:18.662418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.662390 4948 generic.go:356] "Generic (PLEG): container finished" podID="c9f061e8-d50d-4099-b988-3cc62387009d" containerID="125c7d122a2dc7722297558f452a3d5c1a0dd1555df12adda14780d086a1e238" exitCode=0 Nov 03 19:15:18.662658 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.662495 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" event={"ID":"c9f061e8-d50d-4099-b988-3cc62387009d","Type":"ContainerDied","Data":"125c7d122a2dc7722297558f452a3d5c1a0dd1555df12adda14780d086a1e238"} Nov 03 19:15:18.667052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.666359 4948 generic.go:356] "Generic (PLEG): container finished" podID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerID="b394f23eb01685994f3e864793af5b7f7cfec30968aba65fdcbfc099859d8ccf" exitCode=60 Nov 03 19:15:18.679613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.676991 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerDied","Data":"b394f23eb01685994f3e864793af5b7f7cfec30968aba65fdcbfc099859d8ccf"} Nov 03 19:15:18.679613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.677495 4948 scope.go:122] "RemoveContainer" containerID="b394f23eb01685994f3e864793af5b7f7cfec30968aba65fdcbfc099859d8ccf" Nov 03 19:15:18.679613 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:18.677932 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr_odh-ai-gateway-infra(2cb3eff7-234f-4b79-9e31-69e75efc0fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" Nov 03 19:15:18.694759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.683184 4948 scope.go:122] "RemoveContainer" containerID="d2d9e4bae2067d1c2257358f6e32f559aacf6aebabb45bbca552a29e988f9422" Nov 03 19:15:18.703561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.703534 4948 generic.go:356] "Generic (PLEG): container finished" podID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerID="98983e2595f29104dfcc677015638f70c82dea8ea661a7ebd344a8ca8170d123" exitCode=60 Nov 03 19:15:18.703716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.703610 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerDied","Data":"98983e2595f29104dfcc677015638f70c82dea8ea661a7ebd344a8ca8170d123"} Nov 03 19:15:18.704172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.704154 4948 scope.go:122] "RemoveContainer" containerID="98983e2595f29104dfcc677015638f70c82dea8ea661a7ebd344a8ca8170d123" Nov 03 19:15:18.704593 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:18.704571 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f_odh-ai-gateway-infra(32555649-8e98-4fb1-a9a6-29b5a9d6b90e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" Nov 03 19:15:18.718427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.718402 4948 scope.go:122] "RemoveContainer" containerID="c5ec49311fb440cc96fe423fd4fb3ecf680b2d904bffc77c1fcd7ebfbc4440b4" Nov 03 19:15:18.746877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.746833 4948 scope.go:122] "RemoveContainer" containerID="f5333eadbcefc340c3e7b4221b10c73d85dbec76a58044bf2c23ce8ac1c246bb" Nov 03 19:15:18.747124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.747102 4948 scope.go:122] "RemoveContainer" containerID="dc5fd943007ca7c8f9ddd94dc34716c99c74b7710006be79cbce8f9c3042b9ab" Nov 03 19:15:18.747508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.747467 4948 scope.go:122] "RemoveContainer" containerID="155cccc6b65b82488d71bec3e056f2edd82443c452c88940d86f63ea947263ae" Nov 03 19:15:18.748542 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.747780 4948 scope.go:122] "RemoveContainer" containerID="fe6230b0031d1b1d150979ab316dba8dff3db4e2c0b5d0ee0c88176976974c75" Nov 03 19:15:18.760272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.760229 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" containerID="cri-o://58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb" gracePeriod=45 Nov 03 19:15:18.789437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.789377 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" containerID="cri-o://9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1" gracePeriod=45 Nov 03 19:15:18.810421 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.810372 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": read tcp 10.132.0.2:53492->10.132.0.115:8000: read: connection reset by peer" Nov 03 19:15:18.824208 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.824176 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 19:15:18.897539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:18.897505 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr"] Nov 03 19:15:19.140617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.139045 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r"] Nov 03 19:15:19.149080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.149041 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ac09-29895555-nmc2r"] Nov 03 19:15:19.225522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.225496 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:15:19.228871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.228846 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:15:19.270018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.269986 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9afb714-a01e-4660-946a-71ae6568adaa-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afb714-a01e-4660-946a-71ae6568adaa-tls-certs\") pod \"c9afb714-a01e-4660-946a-71ae6568adaa\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " Nov 03 19:15:19.270151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270038 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01e3f68-46a3-4088-8aab-31f0a9e86108-kube-api-access-jv8cg\" (UniqueName: \"kubernetes.io/projected/c01e3f68-46a3-4088-8aab-31f0a9e86108-kube-api-access-jv8cg\") pod \"c01e3f68-46a3-4088-8aab-31f0a9e86108\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " Nov 03 19:15:19.270151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270124 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-tmp-dir\") pod \"c9afb714-a01e-4660-946a-71ae6568adaa\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " Nov 03 19:15:19.270241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270153 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-home\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-home\") pod \"c01e3f68-46a3-4088-8aab-31f0a9e86108\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " Nov 03 19:15:19.270241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270212 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-model-cache\") pod \"c01e3f68-46a3-4088-8aab-31f0a9e86108\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " Nov 03 19:15:19.270314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270271 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-home\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-home\") pod \"c9afb714-a01e-4660-946a-71ae6568adaa\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " Nov 03 19:15:19.270350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270327 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-dshm\") pod \"c01e3f68-46a3-4088-8aab-31f0a9e86108\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " Nov 03 19:15:19.270391 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270376 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-model-cache\") pod \"c9afb714-a01e-4660-946a-71ae6568adaa\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " Nov 03 19:15:19.270433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270420 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-tmp-dir\") pod \"c01e3f68-46a3-4088-8aab-31f0a9e86108\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " Nov 03 19:15:19.270511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270490 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-dshm\") pod \"c9afb714-a01e-4660-946a-71ae6568adaa\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " Nov 03 19:15:19.270558 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270545 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01e3f68-46a3-4088-8aab-31f0a9e86108-tls-certs\" (UniqueName: \"kubernetes.io/secret/c01e3f68-46a3-4088-8aab-31f0a9e86108-tls-certs\") pod \"c01e3f68-46a3-4088-8aab-31f0a9e86108\" (UID: \"c01e3f68-46a3-4088-8aab-31f0a9e86108\") " Nov 03 19:15:19.270615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.270581 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9afb714-a01e-4660-946a-71ae6568adaa-kube-api-access-swzvz\" (UniqueName: \"kubernetes.io/projected/c9afb714-a01e-4660-946a-71ae6568adaa-kube-api-access-swzvz\") pod \"c9afb714-a01e-4660-946a-71ae6568adaa\" (UID: \"c9afb714-a01e-4660-946a-71ae6568adaa\") " Nov 03 19:15:19.273947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.272355 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-home" pod "c01e3f68-46a3-4088-8aab-31f0a9e86108" (UID: "c01e3f68-46a3-4088-8aab-31f0a9e86108"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.273947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.272545 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-model-cache" pod "c01e3f68-46a3-4088-8aab-31f0a9e86108" (UID: "c01e3f68-46a3-4088-8aab-31f0a9e86108"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.273947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.273656 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-home" pod "c9afb714-a01e-4660-946a-71ae6568adaa" (UID: "c9afb714-a01e-4660-946a-71ae6568adaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.273947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.273755 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-tmp-dir" pod "c9afb714-a01e-4660-946a-71ae6568adaa" (UID: "c9afb714-a01e-4660-946a-71ae6568adaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.274321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.274151 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-model-cache" pod "c9afb714-a01e-4660-946a-71ae6568adaa" (UID: "c9afb714-a01e-4660-946a-71ae6568adaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.274731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.274636 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-tmp-dir" pod "c01e3f68-46a3-4088-8aab-31f0a9e86108" (UID: "c01e3f68-46a3-4088-8aab-31f0a9e86108"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.380858 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9afb714-a01e-4660-946a-71ae6568adaa-tls-certs" pod "c9afb714-a01e-4660-946a-71ae6568adaa" (UID: "c9afb714-a01e-4660-946a-71ae6568adaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.382566 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9afb714-a01e-4660-946a-71ae6568adaa-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.382595 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.382608 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.382621 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.382634 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.382649 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.387760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.382662 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.450918 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.450771 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f"] Nov 03 19:15:19.579209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.579036 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9afb714-a01e-4660-946a-71ae6568adaa-kube-api-access-swzvz" pod "c9afb714-a01e-4660-946a-71ae6568adaa" (UID: "c9afb714-a01e-4660-946a-71ae6568adaa"). InnerVolumeSpecName "kube-api-access-swzvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:19.579209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.579174 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-dshm" pod "c9afb714-a01e-4660-946a-71ae6568adaa" (UID: "c9afb714-a01e-4660-946a-71ae6568adaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.620007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.588148 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-dshm" pod "c01e3f68-46a3-4088-8aab-31f0a9e86108" (UID: "c01e3f68-46a3-4088-8aab-31f0a9e86108"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.620007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.589370 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01e3f68-46a3-4088-8aab-31f0a9e86108-tls-certs" pod "c01e3f68-46a3-4088-8aab-31f0a9e86108" (UID: "c01e3f68-46a3-4088-8aab-31f0a9e86108"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:19.620007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.598223 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01e3f68-46a3-4088-8aab-31f0a9e86108-kube-api-access-jv8cg" pod "c01e3f68-46a3-4088-8aab-31f0a9e86108" (UID: "c01e3f68-46a3-4088-8aab-31f0a9e86108"). InnerVolumeSpecName "kube-api-access-jv8cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:19.654790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.642358 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9afb714-a01e-4660-946a-71ae6568adaa-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.654790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.642395 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01e3f68-46a3-4088-8aab-31f0a9e86108-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.654790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.642411 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzvz\" (UniqueName: \"kubernetes.io/projected/c9afb714-a01e-4660-946a-71ae6568adaa-kube-api-access-swzvz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.654790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.642426 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv8cg\" (UniqueName: \"kubernetes.io/projected/c01e3f68-46a3-4088-8aab-31f0a9e86108-kube-api-access-jv8cg\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.654790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.642438 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01e3f68-46a3-4088-8aab-31f0a9e86108-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.768500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.768320 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" podUID="66069f67-85da-41fc-8868-2ddec44a172a" containerName="payload-processing" containerID="cri-o://8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e" gracePeriod=25 Nov 03 19:15:19.901729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.889773 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" podUID="6534ac32-568f-4564-9bd8-3e1b9d190fdc" containerName="payload-pre-processing" containerID="cri-o://414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d" gracePeriod=25 Nov 03 19:15:19.975762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.972321 4948 generic.go:356] "Generic (PLEG): container finished" podID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerID="9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1" exitCode=0 Nov 03 19:15:19.975762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.972548 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" event={"ID":"c01e3f68-46a3-4088-8aab-31f0a9e86108","Type":"ContainerDied","Data":"9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1"} Nov 03 19:15:19.975762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.972585 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" event={"ID":"c01e3f68-46a3-4088-8aab-31f0a9e86108","Type":"ContainerDied","Data":"049242e6ed48f4360dbf2cd94de78eb7a40f812f978c8d3697e1f0b56ec34cea"} Nov 03 19:15:19.975762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.972605 4948 scope.go:122] "RemoveContainer" containerID="9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1" Nov 03 19:15:19.975762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.972768 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv" Nov 03 19:15:19.982811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.982451 4948 generic.go:356] "Generic (PLEG): container finished" podID="c9afb714-a01e-4660-946a-71ae6568adaa" containerID="58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb" exitCode=0 Nov 03 19:15:19.982811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.982606 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" event={"ID":"c9afb714-a01e-4660-946a-71ae6568adaa","Type":"ContainerDied","Data":"58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb"} Nov 03 19:15:19.982811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.982646 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" event={"ID":"c9afb714-a01e-4660-946a-71ae6568adaa","Type":"ContainerDied","Data":"165b46f8b47dbc84bbcb845d692e30f394bc8dcf2693ef615e5933efae3dcc5a"} Nov 03 19:15:19.993321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:19.983100 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf" Nov 03 19:15:20.169612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.169233 4948 scope.go:122] "RemoveContainer" containerID="9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1" Nov 03 19:15:20.170883 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:20.170421 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1\": container with ID starting with 9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1 not found: ID does not exist" containerID="9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1" Nov 03 19:15:20.170883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.170468 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1"} err="failed to get container status \"9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1\": rpc error: code = NotFound desc = could not find container \"9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1\": container with ID starting with 9a0a22d03572124201fbc3710c25aa3b5f6be3474b3d26cb2588cfc1038e99f1 not found: ID does not exist" Nov 03 19:15:20.170883 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.170495 4948 scope.go:122] "RemoveContainer" containerID="58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb" Nov 03 19:15:20.235732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.235605 4948 scope.go:122] "RemoveContainer" containerID="58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb" Nov 03 19:15:20.236285 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:20.236209 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb\": container with ID starting with 58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb not found: ID does not exist" containerID="58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb" Nov 03 19:15:20.236285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.236245 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb"} err="failed to get container status \"58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb\": rpc error: code = NotFound desc = could not find container \"58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb\": container with ID starting with 58bbf9b29f03722c505fefa800a1ffd3e7ee0f1c2a2b5c9703fffb9707dd59eb not found: ID does not exist" Nov 03 19:15:20.312586 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.312554 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:20.325606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.325577 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv"] Nov 03 19:15:20.334657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.334623 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/premium-w6-fba266-kserve-79bbcfb6d9-26cqv"] Nov 03 19:15:20.423918 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.420060 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb3eff7-234f-4b79-9e31-69e75efc0fce-kube-api-access-gkjhh\" (UniqueName: \"kubernetes.io/projected/2cb3eff7-234f-4b79-9e31-69e75efc0fce-kube-api-access-gkjhh\") pod \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " Nov 03 19:15:20.423918 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.420220 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb3eff7-234f-4b79-9e31-69e75efc0fce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb3eff7-234f-4b79-9e31-69e75efc0fce-maas-api-service-ca\") pod \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\" (UID: \"2cb3eff7-234f-4b79-9e31-69e75efc0fce\") " Nov 03 19:15:20.423918 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.423821 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb3eff7-234f-4b79-9e31-69e75efc0fce-maas-api-service-ca" pod "2cb3eff7-234f-4b79-9e31-69e75efc0fce" (UID: "2cb3eff7-234f-4b79-9e31-69e75efc0fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:20.429408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.428123 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb3eff7-234f-4b79-9e31-69e75efc0fce-kube-api-access-gkjhh" pod "2cb3eff7-234f-4b79-9e31-69e75efc0fce" (UID: "2cb3eff7-234f-4b79-9e31-69e75efc0fce"). InnerVolumeSpecName "kube-api-access-gkjhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:20.433602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.433502 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:20.442929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.442837 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:20.462427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.462360 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf"] Nov 03 19:15:20.470479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.470429 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-2-w6-fba266-kserve-5bb7c6b9ff-cfjvf"] Nov 03 19:15:20.522190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.522158 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-openshift-ca\") pod \"c9f061e8-d50d-4099-b988-3cc62387009d\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " Nov 03 19:15:20.522365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.522296 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f061e8-d50d-4099-b988-3cc62387009d-kube-api-access-ps4hd\" (UniqueName: \"kubernetes.io/projected/c9f061e8-d50d-4099-b988-3cc62387009d-kube-api-access-ps4hd\") pod \"c9f061e8-d50d-4099-b988-3cc62387009d\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " Nov 03 19:15:20.522427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.522406 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-maas-api-service-ca\") pod \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " Nov 03 19:15:20.522465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.522451 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-xks-ca\") pod \"c9f061e8-d50d-4099-b988-3cc62387009d\" (UID: \"c9f061e8-d50d-4099-b988-3cc62387009d\") " Nov 03 19:15:20.522553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.522526 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-openshift-ca" pod "c9f061e8-d50d-4099-b988-3cc62387009d" (UID: "c9f061e8-d50d-4099-b988-3cc62387009d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:20.522607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.522540 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-kube-api-access-27c8z\" (UniqueName: \"kubernetes.io/projected/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-kube-api-access-27c8z\") pod \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\" (UID: \"32555649-8e98-4fb1-a9a6-29b5a9d6b90e\") " Nov 03 19:15:20.522837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.522813 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-maas-api-service-ca" pod "32555649-8e98-4fb1-a9a6-29b5a9d6b90e" (UID: "32555649-8e98-4fb1-a9a6-29b5a9d6b90e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:20.523313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.523289 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.523313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.523314 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkjhh\" (UniqueName: \"kubernetes.io/projected/2cb3eff7-234f-4b79-9e31-69e75efc0fce-kube-api-access-gkjhh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.523471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.523324 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb3eff7-234f-4b79-9e31-69e75efc0fce-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.523471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.523333 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.562942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.560200 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-xks-ca" pod "c9f061e8-d50d-4099-b988-3cc62387009d" (UID: "c9f061e8-d50d-4099-b988-3cc62387009d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:20.562942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.560527 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-kube-api-access-27c8z" pod "32555649-8e98-4fb1-a9a6-29b5a9d6b90e" (UID: "32555649-8e98-4fb1-a9a6-29b5a9d6b90e"). InnerVolumeSpecName "kube-api-access-27c8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:20.562942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.561724 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f061e8-d50d-4099-b988-3cc62387009d-kube-api-access-ps4hd" pod "c9f061e8-d50d-4099-b988-3cc62387009d" (UID: "c9f061e8-d50d-4099-b988-3cc62387009d"). InnerVolumeSpecName "kube-api-access-ps4hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:20.624935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.624897 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f061e8-d50d-4099-b988-3cc62387009d-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.625135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.624939 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27c8z\" (UniqueName: \"kubernetes.io/projected/32555649-8e98-4fb1-a9a6-29b5a9d6b90e-kube-api-access-27c8z\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.625135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.624962 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps4hd\" (UniqueName: \"kubernetes.io/projected/c9f061e8-d50d-4099-b988-3cc62387009d-kube-api-access-ps4hd\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.713743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.713714 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:15:20.719187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.719159 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:15:20.726416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.726384 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66069f67-85da-41fc-8868-2ddec44a172a-kube-api-access-ncwck\" (UniqueName: \"kubernetes.io/projected/66069f67-85da-41fc-8868-2ddec44a172a-kube-api-access-ncwck\") pod \"66069f67-85da-41fc-8868-2ddec44a172a\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " Nov 03 19:15:20.726576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.726540 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66069f67-85da-41fc-8868-2ddec44a172a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66069f67-85da-41fc-8868-2ddec44a172a-plugins-config-volume\") pod \"66069f67-85da-41fc-8868-2ddec44a172a\" (UID: \"66069f67-85da-41fc-8868-2ddec44a172a\") " Nov 03 19:15:20.726926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.726901 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66069f67-85da-41fc-8868-2ddec44a172a-plugins-config-volume" pod "66069f67-85da-41fc-8868-2ddec44a172a" (UID: "66069f67-85da-41fc-8868-2ddec44a172a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:20.727176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.727160 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66069f67-85da-41fc-8868-2ddec44a172a-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.728412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.728392 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66069f67-85da-41fc-8868-2ddec44a172a-kube-api-access-ncwck" pod "66069f67-85da-41fc-8868-2ddec44a172a" (UID: "66069f67-85da-41fc-8868-2ddec44a172a"). InnerVolumeSpecName "kube-api-access-ncwck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:20.763437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.763397 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" path="/var/lib/kubelet/pods/8cb756b1-a9d3-4235-980d-d9009a6226ac/volumes" Nov 03 19:15:20.764239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.764216 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" path="/var/lib/kubelet/pods/c01e3f68-46a3-4088-8aab-31f0a9e86108/volumes" Nov 03 19:15:20.764960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.764903 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" path="/var/lib/kubelet/pods/c9afb714-a01e-4660-946a-71ae6568adaa/volumes" Nov 03 19:15:20.828289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.828201 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6534ac32-568f-4564-9bd8-3e1b9d190fdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534ac32-568f-4564-9bd8-3e1b9d190fdc-plugins-config-volume\") pod \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " Nov 03 19:15:20.828779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.828362 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6534ac32-568f-4564-9bd8-3e1b9d190fdc-kube-api-access-tv64g\" (UniqueName: \"kubernetes.io/projected/6534ac32-568f-4564-9bd8-3e1b9d190fdc-kube-api-access-tv64g\") pod \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\" (UID: \"6534ac32-568f-4564-9bd8-3e1b9d190fdc\") " Nov 03 19:15:20.828779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.828563 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6534ac32-568f-4564-9bd8-3e1b9d190fdc-plugins-config-volume" pod "6534ac32-568f-4564-9bd8-3e1b9d190fdc" (UID: "6534ac32-568f-4564-9bd8-3e1b9d190fdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:20.829114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.829092 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwck\" (UniqueName: \"kubernetes.io/projected/66069f67-85da-41fc-8868-2ddec44a172a-kube-api-access-ncwck\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.829230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.829117 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534ac32-568f-4564-9bd8-3e1b9d190fdc-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.830745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.830690 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6534ac32-568f-4564-9bd8-3e1b9d190fdc-kube-api-access-tv64g" pod "6534ac32-568f-4564-9bd8-3e1b9d190fdc" (UID: "6534ac32-568f-4564-9bd8-3e1b9d190fdc"). InnerVolumeSpecName "kube-api-access-tv64g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:20.932924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:20.932888 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv64g\" (UniqueName: \"kubernetes.io/projected/6534ac32-568f-4564-9bd8-3e1b9d190fdc-kube-api-access-tv64g\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.010092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.010057 4948 generic.go:356] "Generic (PLEG): container finished" podID="510004c0-3875-4da4-af50-af656c67668f" containerID="58e29dcb9deabe33cbbaa139f0c346ced46ed7bbdf1c02511312d223c46a7410" exitCode=60 Nov 03 19:15:21.010293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.010096 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" event={"ID":"510004c0-3875-4da4-af50-af656c67668f","Type":"ContainerDied","Data":"58e29dcb9deabe33cbbaa139f0c346ced46ed7bbdf1c02511312d223c46a7410"} Nov 03 19:15:21.010293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.010186 4948 scope.go:122] "RemoveContainer" containerID="f5333eadbcefc340c3e7b4221b10c73d85dbec76a58044bf2c23ce8ac1c246bb" Nov 03 19:15:21.010730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.010692 4948 scope.go:122] "RemoveContainer" containerID="58e29dcb9deabe33cbbaa139f0c346ced46ed7bbdf1c02511312d223c46a7410" Nov 03 19:15:21.011185 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:21.011152 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5_odh-ai-gateway-infra(510004c0-3875-4da4-af50-af656c67668f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" podUID="510004c0-3875-4da4-af50-af656c67668f" Nov 03 19:15:21.013387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.013360 4948 generic.go:356] "Generic (PLEG): container finished" podID="66069f67-85da-41fc-8868-2ddec44a172a" containerID="8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e" exitCode=0 Nov 03 19:15:21.013489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.013422 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" Nov 03 19:15:21.013489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.013439 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" event={"ID":"66069f67-85da-41fc-8868-2ddec44a172a","Type":"ContainerDied","Data":"8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e"} Nov 03 19:15:21.013489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.013478 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc" event={"ID":"66069f67-85da-41fc-8868-2ddec44a172a","Type":"ContainerDied","Data":"6728c0e9d2e6ce58dbbd523bdfd993084dc6f43430621dc073ebe321b53f1016"} Nov 03 19:15:21.017334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.017313 4948 generic.go:356] "Generic (PLEG): container finished" podID="961a3275-52b1-443b-adcc-43af86bddcef" containerID="70d20f9c34de82bcfdf4c2e6f22e988e52f3de258f62e6fac170ed79be2f2772" exitCode=60 Nov 03 19:15:21.017450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.017408 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" event={"ID":"961a3275-52b1-443b-adcc-43af86bddcef","Type":"ContainerDied","Data":"70d20f9c34de82bcfdf4c2e6f22e988e52f3de258f62e6fac170ed79be2f2772"} Nov 03 19:15:21.017870 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.017849 4948 scope.go:122] "RemoveContainer" containerID="70d20f9c34de82bcfdf4c2e6f22e988e52f3de258f62e6fac170ed79be2f2772" Nov 03 19:15:21.018218 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:21.018193 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w_odh-ai-gateway-infra(961a3275-52b1-443b-adcc-43af86bddcef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" podUID="961a3275-52b1-443b-adcc-43af86bddcef" Nov 03 19:15:21.020768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.020738 4948 generic.go:356] "Generic (PLEG): container finished" podID="6534ac32-568f-4564-9bd8-3e1b9d190fdc" containerID="414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d" exitCode=0 Nov 03 19:15:21.020768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.020763 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" Nov 03 19:15:21.020958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.020831 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" event={"ID":"6534ac32-568f-4564-9bd8-3e1b9d190fdc","Type":"ContainerDied","Data":"414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d"} Nov 03 19:15:21.020958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.020873 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8" event={"ID":"6534ac32-568f-4564-9bd8-3e1b9d190fdc","Type":"ContainerDied","Data":"4824cd6e59c6117d722e6ecdbf1e3d7b873fb185a4a9025a097cac655d05da5e"} Nov 03 19:15:21.023431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.023406 4948 generic.go:356] "Generic (PLEG): container finished" podID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerID="c1d838e79e75aec36c9aaa21ec192df2f4ec6703514e0badc66b7eab98ea2cdf" exitCode=60 Nov 03 19:15:21.023540 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.023480 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" event={"ID":"b77fb6c3-2ae1-4eed-8883-d39ddc4da728","Type":"ContainerDied","Data":"c1d838e79e75aec36c9aaa21ec192df2f4ec6703514e0badc66b7eab98ea2cdf"} Nov 03 19:15:21.023967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.023945 4948 scope.go:122] "RemoveContainer" containerID="c1d838e79e75aec36c9aaa21ec192df2f4ec6703514e0badc66b7eab98ea2cdf" Nov 03 19:15:21.024300 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:21.024273 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf_odh-ai-gateway-infra(b77fb6c3-2ae1-4eed-8883-d39ddc4da728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" Nov 03 19:15:21.025160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.025140 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" Nov 03 19:15:21.025249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.025137 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5" event={"ID":"c9f061e8-d50d-4099-b988-3cc62387009d","Type":"ContainerDied","Data":"24f9544c16d7cdbbd4fc2b5573a8deb3b78e0edf265c225a2cb189e0457e1669"} Nov 03 19:15:21.025328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.025262 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24f9544c16d7cdbbd4fc2b5573a8deb3b78e0edf265c225a2cb189e0457e1669" Nov 03 19:15:21.026902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.026880 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" Nov 03 19:15:21.027490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.026883 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr" event={"ID":"2cb3eff7-234f-4b79-9e31-69e75efc0fce","Type":"ContainerDied","Data":"bebe2ba510cbeb559f02d1c496415c1c0859afc542a525b40c781f4059b196bd"} Nov 03 19:15:21.029520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.029495 4948 generic.go:356] "Generic (PLEG): container finished" podID="afb51c78-b827-447d-9603-8d18eb55fed1" containerID="d3356153299dd3a1b2d4adb76599f1ff3fbce5adcacf67df2888ce73356ddf47" exitCode=60 Nov 03 19:15:21.029636 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.029568 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" event={"ID":"afb51c78-b827-447d-9603-8d18eb55fed1","Type":"ContainerDied","Data":"d3356153299dd3a1b2d4adb76599f1ff3fbce5adcacf67df2888ce73356ddf47"} Nov 03 19:15:21.030778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.030027 4948 scope.go:122] "RemoveContainer" containerID="d3356153299dd3a1b2d4adb76599f1ff3fbce5adcacf67df2888ce73356ddf47" Nov 03 19:15:21.030778 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:21.030335 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz_odh-ai-gateway-infra(afb51c78-b827-447d-9603-8d18eb55fed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" Nov 03 19:15:21.030778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.030449 4948 scope.go:122] "RemoveContainer" containerID="8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e" Nov 03 19:15:21.032934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.032912 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" event={"ID":"32555649-8e98-4fb1-a9a6-29b5a9d6b90e","Type":"ContainerDied","Data":"fa37b97f677b5a8f38eb3c6a6636c64f6cc7114bce8c0a5aaecfce8204eed432"} Nov 03 19:15:21.033028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.033014 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f" Nov 03 19:15:21.045084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.044945 4948 scope.go:122] "RemoveContainer" containerID="8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e" Nov 03 19:15:21.045367 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:21.045339 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e\": container with ID starting with 8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e not found: ID does not exist" containerID="8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e" Nov 03 19:15:21.045448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.045377 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e"} err="failed to get container status \"8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e\": rpc error: code = NotFound desc = could not find container \"8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e\": container with ID starting with 8a42bc48ff1c10ed003b5446d36570e1438e667a3051e150a5c95bd9dc7e259e not found: ID does not exist" Nov 03 19:15:21.045448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.045402 4948 scope.go:122] "RemoveContainer" containerID="fe6230b0031d1b1d150979ab316dba8dff3db4e2c0b5d0ee0c88176976974c75" Nov 03 19:15:21.066659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.066596 4948 scope.go:122] "RemoveContainer" containerID="414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d" Nov 03 19:15:21.078344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.078299 4948 scope.go:122] "RemoveContainer" containerID="414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d" Nov 03 19:15:21.078683 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:21.078656 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d\": container with ID starting with 414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d not found: ID does not exist" containerID="414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d" Nov 03 19:15:21.078775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.078693 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d"} err="failed to get container status \"414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d\": rpc error: code = NotFound desc = could not find container \"414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d\": container with ID starting with 414ef08302013ac526813cf31bfd436c183599c4fa07bd91ce51f15bd16ccb4d not found: ID does not exist" Nov 03 19:15:21.078775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.078731 4948 scope.go:122] "RemoveContainer" containerID="dc5fd943007ca7c8f9ddd94dc34716c99c74b7710006be79cbce8f9c3042b9ab" Nov 03 19:15:21.095372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.095351 4948 scope.go:122] "RemoveContainer" containerID="b394f23eb01685994f3e864793af5b7f7cfec30968aba65fdcbfc099859d8ccf" Nov 03 19:15:21.106176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.106153 4948 scope.go:122] "RemoveContainer" containerID="155cccc6b65b82488d71bec3e056f2edd82443c452c88940d86f63ea947263ae" Nov 03 19:15:21.122999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.122975 4948 scope.go:122] "RemoveContainer" containerID="98983e2595f29104dfcc677015638f70c82dea8ea661a7ebd344a8ca8170d123" Nov 03 19:15:21.635896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.635864 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr"] Nov 03 19:15:21.640373 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.640349 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770e67-29895555-6tdlr"] Nov 03 19:15:21.808825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.808790 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8"] Nov 03 19:15:21.813419 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:21.813393 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-97e020-76946549b7-mf7h8"] Nov 03 19:15:22.108978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.108917 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc"] Nov 03 19:15:22.115039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.114998 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-97e020-b5c87c667-vjfmc"] Nov 03 19:15:22.173987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.173952 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f"] Nov 03 19:15:22.179013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.178981 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895555-c5z2f"] Nov 03 19:15:22.298220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.298176 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w"] Nov 03 19:15:22.412294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.412265 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz"] Nov 03 19:15:22.505121 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.504789 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:22.531019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.530984 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf"] Nov 03 19:15:22.634246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.634204 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" podUID="92d4548b-1f30-482a-b681-92323671e3d4" containerName="payload-processing" containerID="cri-o://aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1" gracePeriod=25 Nov 03 19:15:22.642457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.642427 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:22.672916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.672190 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961a3275-52b1-443b-adcc-43af86bddcef-kube-api-access-fnqbl\" (UniqueName: \"kubernetes.io/projected/961a3275-52b1-443b-adcc-43af86bddcef-kube-api-access-fnqbl\") pod \"961a3275-52b1-443b-adcc-43af86bddcef\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " Nov 03 19:15:22.672916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.672248 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961a3275-52b1-443b-adcc-43af86bddcef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961a3275-52b1-443b-adcc-43af86bddcef-maas-api-service-ca\") pod \"961a3275-52b1-443b-adcc-43af86bddcef\" (UID: \"961a3275-52b1-443b-adcc-43af86bddcef\") " Nov 03 19:15:22.677882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.674777 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961a3275-52b1-443b-adcc-43af86bddcef-maas-api-service-ca" pod "961a3275-52b1-443b-adcc-43af86bddcef" (UID: "961a3275-52b1-443b-adcc-43af86bddcef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.684576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.684521 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961a3275-52b1-443b-adcc-43af86bddcef-kube-api-access-fnqbl" pod "961a3275-52b1-443b-adcc-43af86bddcef" (UID: "961a3275-52b1-443b-adcc-43af86bddcef"). InnerVolumeSpecName "kube-api-access-fnqbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.761280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.761240 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" path="/var/lib/kubelet/pods/2cb3eff7-234f-4b79-9e31-69e75efc0fce/volumes" Nov 03 19:15:22.762358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.762332 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" path="/var/lib/kubelet/pods/32555649-8e98-4fb1-a9a6-29b5a9d6b90e/volumes" Nov 03 19:15:22.762977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.762954 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6534ac32-568f-4564-9bd8-3e1b9d190fdc" path="/var/lib/kubelet/pods/6534ac32-568f-4564-9bd8-3e1b9d190fdc/volumes" Nov 03 19:15:22.767793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.763931 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66069f67-85da-41fc-8868-2ddec44a172a" path="/var/lib/kubelet/pods/66069f67-85da-41fc-8868-2ddec44a172a/volumes" Nov 03 19:15:22.770290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.770209 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:22.773862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.773841 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb51c78-b827-447d-9603-8d18eb55fed1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb51c78-b827-447d-9603-8d18eb55fed1-maas-api-service-ca\") pod \"afb51c78-b827-447d-9603-8d18eb55fed1\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " Nov 03 19:15:22.774010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.773884 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb51c78-b827-447d-9603-8d18eb55fed1-kube-api-access-m4fgh\" (UniqueName: \"kubernetes.io/projected/afb51c78-b827-447d-9603-8d18eb55fed1-kube-api-access-m4fgh\") pod \"afb51c78-b827-447d-9603-8d18eb55fed1\" (UID: \"afb51c78-b827-447d-9603-8d18eb55fed1\") " Nov 03 19:15:22.774361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.774342 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnqbl\" (UniqueName: \"kubernetes.io/projected/961a3275-52b1-443b-adcc-43af86bddcef-kube-api-access-fnqbl\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.774432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.774362 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961a3275-52b1-443b-adcc-43af86bddcef-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.774637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.774612 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb51c78-b827-447d-9603-8d18eb55fed1-maas-api-service-ca" pod "afb51c78-b827-447d-9603-8d18eb55fed1" (UID: "afb51c78-b827-447d-9603-8d18eb55fed1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.777584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.777557 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb51c78-b827-447d-9603-8d18eb55fed1-kube-api-access-m4fgh" pod "afb51c78-b827-447d-9603-8d18eb55fed1" (UID: "afb51c78-b827-447d-9603-8d18eb55fed1"). InnerVolumeSpecName "kube-api-access-m4fgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.799684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.799632 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" podUID="e47fa162-17a9-4f0a-ad54-8ee6add5071d" containerName="payload-pre-processing" containerID="cri-o://151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493" gracePeriod=25 Nov 03 19:15:22.875081 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.875045 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-maas-api-service-ca\") pod \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " Nov 03 19:15:22.875290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.875267 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-kube-api-access-9b7qr\" (UniqueName: \"kubernetes.io/projected/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-kube-api-access-9b7qr\") pod \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\" (UID: \"b77fb6c3-2ae1-4eed-8883-d39ddc4da728\") " Nov 03 19:15:22.877555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.875729 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-maas-api-service-ca" pod "b77fb6c3-2ae1-4eed-8883-d39ddc4da728" (UID: "b77fb6c3-2ae1-4eed-8883-d39ddc4da728"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.877555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.877523 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb51c78-b827-447d-9603-8d18eb55fed1-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.877555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.877547 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4fgh\" (UniqueName: \"kubernetes.io/projected/afb51c78-b827-447d-9603-8d18eb55fed1-kube-api-access-m4fgh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.877890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.877562 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.880784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.880754 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-kube-api-access-9b7qr" pod "b77fb6c3-2ae1-4eed-8883-d39ddc4da728" (UID: "b77fb6c3-2ae1-4eed-8883-d39ddc4da728"). InnerVolumeSpecName "kube-api-access-9b7qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:22.932562 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.932532 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 19:15:22.938218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.938187 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5"] Nov 03 19:15:22.979840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.979809 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d4548b-1f30-482a-b681-92323671e3d4-kube-api-access-fbrq5\" (UniqueName: \"kubernetes.io/projected/92d4548b-1f30-482a-b681-92323671e3d4-kube-api-access-fbrq5\") pod \"92d4548b-1f30-482a-b681-92323671e3d4\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " Nov 03 19:15:22.980038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.980027 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d4548b-1f30-482a-b681-92323671e3d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d4548b-1f30-482a-b681-92323671e3d4-plugins-config-volume\") pod \"92d4548b-1f30-482a-b681-92323671e3d4\" (UID: \"92d4548b-1f30-482a-b681-92323671e3d4\") " Nov 03 19:15:22.980896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.980868 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d4548b-1f30-482a-b681-92323671e3d4-plugins-config-volume" pod "92d4548b-1f30-482a-b681-92323671e3d4" (UID: "92d4548b-1f30-482a-b681-92323671e3d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:22.981308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.981280 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d4548b-1f30-482a-b681-92323671e3d4-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.981384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.981313 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7qr\" (UniqueName: \"kubernetes.io/projected/b77fb6c3-2ae1-4eed-8883-d39ddc4da728-kube-api-access-9b7qr\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.983198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:22.983170 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d4548b-1f30-482a-b681-92323671e3d4-kube-api-access-fbrq5" pod "92d4548b-1f30-482a-b681-92323671e3d4" (UID: "92d4548b-1f30-482a-b681-92323671e3d4"). InnerVolumeSpecName "kube-api-access-fbrq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:23.088718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.088667 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbrq5\" (UniqueName: \"kubernetes.io/projected/92d4548b-1f30-482a-b681-92323671e3d4-kube-api-access-fbrq5\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:23.125600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.125563 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 19:15:23.148968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.148928 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:23.151391 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.151362 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" event={"ID":"b77fb6c3-2ae1-4eed-8883-d39ddc4da728","Type":"ContainerDied","Data":"245a4613bf2b4983b78946d17e0384f12a67c50524fa93569d3f161b6f1fde45"} Nov 03 19:15:23.151519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.151437 4948 scope.go:122] "RemoveContainer" containerID="c1d838e79e75aec36c9aaa21ec192df2f4ec6703514e0badc66b7eab98ea2cdf" Nov 03 19:15:23.151601 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.151550 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf" Nov 03 19:15:23.156558 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.156535 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" Nov 03 19:15:23.156657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.156578 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz" event={"ID":"afb51c78-b827-447d-9603-8d18eb55fed1","Type":"ContainerDied","Data":"c4e385274571fe2b6157c84f39371bdd99e9b080fdb68ce07d195c8459794afe"} Nov 03 19:15:23.158556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.158526 4948 generic.go:356] "Generic (PLEG): container finished" podID="e47fa162-17a9-4f0a-ad54-8ee6add5071d" containerID="151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493" exitCode=0 Nov 03 19:15:23.158684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.158569 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" Nov 03 19:15:23.158793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.158724 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" event={"ID":"e47fa162-17a9-4f0a-ad54-8ee6add5071d","Type":"ContainerDied","Data":"151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493"} Nov 03 19:15:23.158793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.158774 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2" event={"ID":"e47fa162-17a9-4f0a-ad54-8ee6add5071d","Type":"ContainerDied","Data":"a8cdebbb8df0d56ffc180f685af85a9667d1e6cf92c3aff10f84551b51534914"} Nov 03 19:15:23.160650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.160631 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" Nov 03 19:15:23.160844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.160822 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w" event={"ID":"961a3275-52b1-443b-adcc-43af86bddcef","Type":"ContainerDied","Data":"44eb32847d9f4d83e1307cadf87dbb213168bc15833c2894dab66f1ce24b06ca"} Nov 03 19:15:23.163912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.163886 4948 generic.go:356] "Generic (PLEG): container finished" podID="92d4548b-1f30-482a-b681-92323671e3d4" containerID="aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1" exitCode=0 Nov 03 19:15:23.164039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.163923 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" event={"ID":"92d4548b-1f30-482a-b681-92323671e3d4","Type":"ContainerDied","Data":"aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1"} Nov 03 19:15:23.164039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.163957 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" event={"ID":"92d4548b-1f30-482a-b681-92323671e3d4","Type":"ContainerDied","Data":"b72cbd0a2dadb97932ba2415a7fe6a516ca143261a3a05f7b6eae8dfcf54650e"} Nov 03 19:15:23.164039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.164016 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f" Nov 03 19:15:23.168825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.168803 4948 scope.go:122] "RemoveContainer" containerID="d3356153299dd3a1b2d4adb76599f1ff3fbce5adcacf67df2888ce73356ddf47" Nov 03 19:15:23.183361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.183337 4948 scope.go:122] "RemoveContainer" containerID="151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493" Nov 03 19:15:23.190122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.190094 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47fa162-17a9-4f0a-ad54-8ee6add5071d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fa162-17a9-4f0a-ad54-8ee6add5071d-plugins-config-volume\") pod \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " Nov 03 19:15:23.190249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.190204 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510004c0-3875-4da4-af50-af656c67668f-kube-api-access-8q9k7\" (UniqueName: \"kubernetes.io/projected/510004c0-3875-4da4-af50-af656c67668f-kube-api-access-8q9k7\") pod \"510004c0-3875-4da4-af50-af656c67668f\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " Nov 03 19:15:23.190345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.190282 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510004c0-3875-4da4-af50-af656c67668f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510004c0-3875-4da4-af50-af656c67668f-maas-api-service-ca\") pod \"510004c0-3875-4da4-af50-af656c67668f\" (UID: \"510004c0-3875-4da4-af50-af656c67668f\") " Nov 03 19:15:23.190345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.190314 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47fa162-17a9-4f0a-ad54-8ee6add5071d-kube-api-access-xhp6z\" (UniqueName: \"kubernetes.io/projected/e47fa162-17a9-4f0a-ad54-8ee6add5071d-kube-api-access-xhp6z\") pod \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\" (UID: \"e47fa162-17a9-4f0a-ad54-8ee6add5071d\") " Nov 03 19:15:23.190660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.190634 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510004c0-3875-4da4-af50-af656c67668f-maas-api-service-ca" pod "510004c0-3875-4da4-af50-af656c67668f" (UID: "510004c0-3875-4da4-af50-af656c67668f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:23.190868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.190839 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47fa162-17a9-4f0a-ad54-8ee6add5071d-plugins-config-volume" pod "e47fa162-17a9-4f0a-ad54-8ee6add5071d" (UID: "e47fa162-17a9-4f0a-ad54-8ee6add5071d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:23.191206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.191179 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47fa162-17a9-4f0a-ad54-8ee6add5071d-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:23.191286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.191214 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510004c0-3875-4da4-af50-af656c67668f-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:23.192604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.192580 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510004c0-3875-4da4-af50-af656c67668f-kube-api-access-8q9k7" pod "510004c0-3875-4da4-af50-af656c67668f" (UID: "510004c0-3875-4da4-af50-af656c67668f"). InnerVolumeSpecName "kube-api-access-8q9k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:23.193276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.193250 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47fa162-17a9-4f0a-ad54-8ee6add5071d-kube-api-access-xhp6z" pod "e47fa162-17a9-4f0a-ad54-8ee6add5071d" (UID: "e47fa162-17a9-4f0a-ad54-8ee6add5071d"). InnerVolumeSpecName "kube-api-access-xhp6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:23.197418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.197394 4948 scope.go:122] "RemoveContainer" containerID="151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493" Nov 03 19:15:23.197885 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:23.197862 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493\": container with ID starting with 151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493 not found: ID does not exist" containerID="151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493" Nov 03 19:15:23.197994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.197893 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493"} err="failed to get container status \"151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493\": rpc error: code = NotFound desc = could not find container \"151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493\": container with ID starting with 151daae48a62550ff323976b6be64fb5a5fae57b4d759b0ebbc9f5e249f29493 not found: ID does not exist" Nov 03 19:15:23.197994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.197910 4948 scope.go:122] "RemoveContainer" containerID="70d20f9c34de82bcfdf4c2e6f22e988e52f3de258f62e6fac170ed79be2f2772" Nov 03 19:15:23.209051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.209031 4948 scope.go:122] "RemoveContainer" containerID="aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1" Nov 03 19:15:23.219203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.219182 4948 scope.go:122] "RemoveContainer" containerID="aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1" Nov 03 19:15:23.219483 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:23.219457 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1\": container with ID starting with aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1 not found: ID does not exist" containerID="aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1" Nov 03 19:15:23.219532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.219485 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1"} err="failed to get container status \"aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1\": rpc error: code = NotFound desc = could not find container \"aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1\": container with ID starting with aedace0c92a9d3f646f888f139419665bf09b48b2da68c2e3d4c91f1d882f9a1 not found: ID does not exist" Nov 03 19:15:23.299726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.292272 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q9k7\" (UniqueName: \"kubernetes.io/projected/510004c0-3875-4da4-af50-af656c67668f-kube-api-access-8q9k7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:23.299726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.292296 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhp6z\" (UniqueName: \"kubernetes.io/projected/e47fa162-17a9-4f0a-ad54-8ee6add5071d-kube-api-access-xhp6z\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:23.502559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.502524 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f"] Nov 03 19:15:23.507006 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.506978 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fba266-56687d67d5-5d54f"] Nov 03 19:15:23.613849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.613814 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w"] Nov 03 19:15:23.618239 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.618214 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a4cacd-29895555-7fp4w"] Nov 03 19:15:23.728989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.728906 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz"] Nov 03 19:15:23.733299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.733272 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba5674-29895555-295xz"] Nov 03 19:15:23.843479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.843443 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf"] Nov 03 19:15:23.848410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:23.848382 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-efd81c-29895555-mtvbf"] Nov 03 19:15:24.082492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.082403 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2"] Nov 03 19:15:24.089584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.089548 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fba266-5df47f6575-qfhx2"] Nov 03 19:15:24.181397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.181367 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" event={"ID":"510004c0-3875-4da4-af50-af656c67668f","Type":"ContainerDied","Data":"2ac06bc73953de93c0d360e3c336ad3357e7aa2f2d1ba841820c9132af711e6e"} Nov 03 19:15:24.181857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.181421 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5" Nov 03 19:15:24.181857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.181427 4948 scope.go:122] "RemoveContainer" containerID="58e29dcb9deabe33cbbaa139f0c346ced46ed7bbdf1c02511312d223c46a7410" Nov 03 19:15:24.354257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.354168 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5"] Nov 03 19:15:24.360729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.360672 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-97e020-29895555-svfs5"] Nov 03 19:15:24.757052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.757018 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510004c0-3875-4da4-af50-af656c67668f" path="/var/lib/kubelet/pods/510004c0-3875-4da4-af50-af656c67668f/volumes" Nov 03 19:15:24.757562 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.757533 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d4548b-1f30-482a-b681-92323671e3d4" path="/var/lib/kubelet/pods/92d4548b-1f30-482a-b681-92323671e3d4/volumes" Nov 03 19:15:24.758003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.757987 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961a3275-52b1-443b-adcc-43af86bddcef" path="/var/lib/kubelet/pods/961a3275-52b1-443b-adcc-43af86bddcef/volumes" Nov 03 19:15:24.758787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.758769 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" path="/var/lib/kubelet/pods/afb51c78-b827-447d-9603-8d18eb55fed1/volumes" Nov 03 19:15:24.759174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.759161 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" path="/var/lib/kubelet/pods/b77fb6c3-2ae1-4eed-8883-d39ddc4da728/volumes" Nov 03 19:15:24.759553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:24.759541 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47fa162-17a9-4f0a-ad54-8ee6add5071d" path="/var/lib/kubelet/pods/e47fa162-17a9-4f0a-ad54-8ee6add5071d/volumes" Nov 03 19:15:25.088531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.088440 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5"] Nov 03 19:15:25.088770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.088737 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" containerID="cri-o://ea555dd5988dfa441f6828aa68093fc36eecdc00030d2c6e2524db91dfe5a12a" gracePeriod=30 Nov 03 19:15:25.102726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.102357 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62"] Nov 03 19:15:25.207732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.200164 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs"] Nov 03 19:15:25.978069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978034 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq"] Nov 03 19:15:25.978816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978798 4948 cpu_manager.go:414] "containerMap: removing container" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:15:25.978869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978816 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:15:25.978869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978834 4948 cpu_manager.go:414] "containerMap: removing container" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:15:25.978869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978839 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:15:25.978869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978847 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" Nov 03 19:15:25.978869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978852 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" Nov 03 19:15:25.978869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978859 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6534ac32-568f-4564-9bd8-3e1b9d190fdc" containerName="payload-pre-processing" Nov 03 19:15:25.978869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978865 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6534ac32-568f-4564-9bd8-3e1b9d190fdc" containerName="payload-pre-processing" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978877 4948 cpu_manager.go:414] "containerMap: removing container" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978881 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978890 4948 cpu_manager.go:414] "containerMap: removing container" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978894 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978902 4948 cpu_manager.go:414] "containerMap: removing container" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978906 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978912 4948 cpu_manager.go:414] "containerMap: removing container" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978916 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978922 4948 cpu_manager.go:414] "containerMap: removing container" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978927 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978936 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e47fa162-17a9-4f0a-ad54-8ee6add5071d" containerName="payload-pre-processing" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978942 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47fa162-17a9-4f0a-ad54-8ee6add5071d" containerName="payload-pre-processing" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978954 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978958 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978967 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978972 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978982 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978986 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978993 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.978997 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979012 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979016 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979029 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7b909622-2a7b-48de-b2cc-50bc64b60f57" containerName="revoke-keys" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979033 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b909622-2a7b-48de-b2cc-50bc64b60f57" containerName="revoke-keys" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979041 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c9f061e8-d50d-4099-b988-3cc62387009d" containerName="revoke-keys" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979046 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f061e8-d50d-4099-b988-3cc62387009d" containerName="revoke-keys" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979054 4948 cpu_manager.go:414] "containerMap: removing container" podUID="92d4548b-1f30-482a-b681-92323671e3d4" containerName="payload-processing" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979059 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d4548b-1f30-482a-b681-92323671e3d4" containerName="payload-processing" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979068 4948 cpu_manager.go:414] "containerMap: removing container" podUID="66069f67-85da-41fc-8868-2ddec44a172a" containerName="payload-processing" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979072 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66069f67-85da-41fc-8868-2ddec44a172a" containerName="payload-processing" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979079 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979083 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979096 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979101 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979109 4948 cpu_manager.go:414] "containerMap: removing container" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:15:25.979098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979114 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979220 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01e3f68-46a3-4088-8aab-31f0a9e86108" containerName="main" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979230 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="6534ac32-568f-4564-9bd8-3e1b9d190fdc" containerName="payload-pre-processing" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979237 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979242 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979247 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979252 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979258 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979264 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47fa162-17a9-4f0a-ad54-8ee6add5071d" containerName="payload-pre-processing" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979271 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979277 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979282 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979287 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979295 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b909622-2a7b-48de-b2cc-50bc64b60f57" containerName="revoke-keys" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979299 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f061e8-d50d-4099-b988-3cc62387009d" containerName="revoke-keys" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979305 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979310 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d4548b-1f30-482a-b681-92323671e3d4" containerName="payload-processing" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979315 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979320 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="66069f67-85da-41fc-8868-2ddec44a172a" containerName="payload-processing" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979325 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9afb714-a01e-4660-946a-71ae6568adaa" containerName="main" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979330 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979337 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979342 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979347 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979351 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:15:25.980191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.979356 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:15:25.987422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:25.987397 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.036352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.036318 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq"] Nov 03 19:15:26.041140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.041108 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:15:26.041313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.041172 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:15:26.078854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.078808 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.079067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.078995 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgns\" (UniqueName: \"kubernetes.io/projected/186aeaf8-9f70-416a-b468-8afc641cfaa4-kube-api-access-7wgns\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.079142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.079110 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.180675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.180630 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.180875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.180816 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgns\" (UniqueName: \"kubernetes.io/projected/186aeaf8-9f70-416a-b468-8afc641cfaa4-kube-api-access-7wgns\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.180924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.180896 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.181607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.181582 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.183306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.183282 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.243299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.243222 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgns\" (UniqueName: \"kubernetes.io/projected/186aeaf8-9f70-416a-b468-8afc641cfaa4-kube-api-access-7wgns\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.307535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.307502 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:26.701167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:26.701130 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq"] Nov 03 19:15:27.319745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:27.319683 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" event={"ID":"186aeaf8-9f70-416a-b468-8afc641cfaa4","Type":"ContainerStarted","Data":"d1bbb846be6ef89160a7d36f501b63da092b8694f903b002460cbc7d61f84a54"} Nov 03 19:15:28.332331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:28.332217 4948 generic.go:356] "Generic (PLEG): container finished" podID="186aeaf8-9f70-416a-b468-8afc641cfaa4" containerID="c26742cf1c5dbd6dd13ab071661826ee54d2fb839fefcd81ba82fb0c521d41e0" exitCode=0 Nov 03 19:15:28.332331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:28.332291 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" event={"ID":"186aeaf8-9f70-416a-b468-8afc641cfaa4","Type":"ContainerDied","Data":"c26742cf1c5dbd6dd13ab071661826ee54d2fb839fefcd81ba82fb0c521d41e0"} Nov 03 19:15:29.521758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.521729 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:29.614563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.614461 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-openshift-ca\") pod \"186aeaf8-9f70-416a-b468-8afc641cfaa4\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " Nov 03 19:15:29.614563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.614519 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186aeaf8-9f70-416a-b468-8afc641cfaa4-kube-api-access-7wgns\" (UniqueName: \"kubernetes.io/projected/186aeaf8-9f70-416a-b468-8afc641cfaa4-kube-api-access-7wgns\") pod \"186aeaf8-9f70-416a-b468-8afc641cfaa4\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " Nov 03 19:15:29.614857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.614675 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-xks-ca\") pod \"186aeaf8-9f70-416a-b468-8afc641cfaa4\" (UID: \"186aeaf8-9f70-416a-b468-8afc641cfaa4\") " Nov 03 19:15:29.614920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.614860 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-openshift-ca" pod "186aeaf8-9f70-416a-b468-8afc641cfaa4" (UID: "186aeaf8-9f70-416a-b468-8afc641cfaa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:29.615280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.615259 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:29.616675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.616650 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-xks-ca" pod "186aeaf8-9f70-416a-b468-8afc641cfaa4" (UID: "186aeaf8-9f70-416a-b468-8afc641cfaa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:29.616899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.616873 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186aeaf8-9f70-416a-b468-8afc641cfaa4-kube-api-access-7wgns" pod "186aeaf8-9f70-416a-b468-8afc641cfaa4" (UID: "186aeaf8-9f70-416a-b468-8afc641cfaa4"). InnerVolumeSpecName "kube-api-access-7wgns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:29.716661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.716621 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wgns\" (UniqueName: \"kubernetes.io/projected/186aeaf8-9f70-416a-b468-8afc641cfaa4-kube-api-access-7wgns\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:29.716661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:29.716652 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186aeaf8-9f70-416a-b468-8afc641cfaa4-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.102783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.102741 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" podUID="be04bddc-7158-41f5-9556-cef105850742" containerName="payload-processing" containerID="cri-o://dafec5197639b65495ca8012285aef277fbe1d84a5e4c71a2750aac8fc9c66ed" gracePeriod=25 Nov 03 19:15:30.193307 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:15:30.192989 4948 logging.go:55] [core] [Channel #7577 SubChannel #7578]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.202:50052", ServerName: "10.132.0.202:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.202:50052: connect: connection refused" Nov 03 19:15:30.201009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.200970 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" podUID="b165948d-4b0c-4df0-b225-3675562ae7e3" containerName="payload-pre-processing" containerID="cri-o://f3f3086442e2725d60e024b0a7f1c83aeddcc3ffe71a505b5057a54ab0575659" gracePeriod=25 Nov 03 19:15:30.368524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.367418 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" event={"ID":"186aeaf8-9f70-416a-b468-8afc641cfaa4","Type":"ContainerDied","Data":"d1bbb846be6ef89160a7d36f501b63da092b8694f903b002460cbc7d61f84a54"} Nov 03 19:15:30.368524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.367605 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq" Nov 03 19:15:30.368814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.368775 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1bbb846be6ef89160a7d36f501b63da092b8694f903b002460cbc7d61f84a54" Nov 03 19:15:30.370533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.370500 4948 generic.go:356] "Generic (PLEG): container finished" podID="b165948d-4b0c-4df0-b225-3675562ae7e3" containerID="f3f3086442e2725d60e024b0a7f1c83aeddcc3ffe71a505b5057a54ab0575659" exitCode=0 Nov 03 19:15:30.370745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.370723 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" event={"ID":"b165948d-4b0c-4df0-b225-3675562ae7e3","Type":"ContainerDied","Data":"f3f3086442e2725d60e024b0a7f1c83aeddcc3ffe71a505b5057a54ab0575659"} Nov 03 19:15:30.373089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.373058 4948 generic.go:356] "Generic (PLEG): container finished" podID="be04bddc-7158-41f5-9556-cef105850742" containerID="dafec5197639b65495ca8012285aef277fbe1d84a5e4c71a2750aac8fc9c66ed" exitCode=0 Nov 03 19:15:30.373231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.373209 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" event={"ID":"be04bddc-7158-41f5-9556-cef105850742","Type":"ContainerDied","Data":"dafec5197639b65495ca8012285aef277fbe1d84a5e4c71a2750aac8fc9c66ed"} Nov 03 19:15:30.376065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.376038 4948 generic.go:356] "Generic (PLEG): container finished" podID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerID="ea555dd5988dfa441f6828aa68093fc36eecdc00030d2c6e2524db91dfe5a12a" exitCode=0 Nov 03 19:15:30.376204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.376186 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" event={"ID":"39803e8e-ba6f-449a-9d53-116d0147e05d","Type":"ContainerDied","Data":"ea555dd5988dfa441f6828aa68093fc36eecdc00030d2c6e2524db91dfe5a12a"} Nov 03 19:15:30.376272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.376243 4948 scope.go:122] "RemoveContainer" containerID="8dc21d0b922a2231a1e793cae12e19ad49ad94ad2f0296117eb0cacd59b1b9db" Nov 03 19:15:30.413300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.412297 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:15:30.428546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.428513 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-data\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.428850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.428828 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-credential-socket\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.429014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.428998 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-socket\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.429245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.429227 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-token\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-token\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.429397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.429382 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-envoy\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.429557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.429540 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39803e8e-ba6f-449a-9d53-116d0147e05d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39803e8e-ba6f-449a-9d53-116d0147e05d-istiod-ca-cert\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.429722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.429687 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-podinfo\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.429907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.429890 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-certs\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.430009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.429981 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-socket" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:30.430105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.430090 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-kube-api-access-gkbnt\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-kube-api-access-gkbnt\") pod \"39803e8e-ba6f-449a-9d53-116d0147e05d\" (UID: \"39803e8e-ba6f-449a-9d53-116d0147e05d\") " Nov 03 19:15:30.430217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.430195 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-data" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:30.430457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.430436 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-credential-socket" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:30.431017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.430993 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.431146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.431130 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.431226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.431216 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.431292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.431012 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39803e8e-ba6f-449a-9d53-116d0147e05d-istiod-ca-cert" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:30.435551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.434264 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-certs" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:30.437385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.437353 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-podinfo" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:30.439519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.439449 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-envoy" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:30.439519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.439497 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-kube-api-access-gkbnt" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "kube-api-access-gkbnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:30.440608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.440580 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-token" pod "39803e8e-ba6f-449a-9d53-116d0147e05d" (UID: "39803e8e-ba6f-449a-9d53-116d0147e05d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:30.470741 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.466934 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:15:30.532208 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532185 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be04bddc-7158-41f5-9556-cef105850742-kube-api-access-ljkzc\" (UniqueName: \"kubernetes.io/projected/be04bddc-7158-41f5-9556-cef105850742-kube-api-access-ljkzc\") pod \"be04bddc-7158-41f5-9556-cef105850742\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " Nov 03 19:15:30.532606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532268 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be04bddc-7158-41f5-9556-cef105850742-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be04bddc-7158-41f5-9556-cef105850742-plugins-config-volume\") pod \"be04bddc-7158-41f5-9556-cef105850742\" (UID: \"be04bddc-7158-41f5-9556-cef105850742\") " Nov 03 19:15:30.532890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532866 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.532988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532892 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.532988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532899 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be04bddc-7158-41f5-9556-cef105850742-plugins-config-volume" pod "be04bddc-7158-41f5-9556-cef105850742" (UID: "be04bddc-7158-41f5-9556-cef105850742"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:30.532988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532908 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39803e8e-ba6f-449a-9d53-116d0147e05d-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.532988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532949 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39803e8e-ba6f-449a-9d53-116d0147e05d-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.532988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532966 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39803e8e-ba6f-449a-9d53-116d0147e05d-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.532988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.532980 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbnt\" (UniqueName: \"kubernetes.io/projected/39803e8e-ba6f-449a-9d53-116d0147e05d-kube-api-access-gkbnt\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.534448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.534425 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be04bddc-7158-41f5-9556-cef105850742-kube-api-access-ljkzc" pod "be04bddc-7158-41f5-9556-cef105850742" (UID: "be04bddc-7158-41f5-9556-cef105850742"). InnerVolumeSpecName "kube-api-access-ljkzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:30.557751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.557694 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:15:30.633980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.633887 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b165948d-4b0c-4df0-b225-3675562ae7e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b165948d-4b0c-4df0-b225-3675562ae7e3-plugins-config-volume\") pod \"b165948d-4b0c-4df0-b225-3675562ae7e3\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " Nov 03 19:15:30.633980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.633967 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b165948d-4b0c-4df0-b225-3675562ae7e3-kube-api-access-lvvsn\" (UniqueName: \"kubernetes.io/projected/b165948d-4b0c-4df0-b225-3675562ae7e3-kube-api-access-lvvsn\") pod \"b165948d-4b0c-4df0-b225-3675562ae7e3\" (UID: \"b165948d-4b0c-4df0-b225-3675562ae7e3\") " Nov 03 19:15:30.634306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.634282 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b165948d-4b0c-4df0-b225-3675562ae7e3-plugins-config-volume" pod "b165948d-4b0c-4df0-b225-3675562ae7e3" (UID: "b165948d-4b0c-4df0-b225-3675562ae7e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:30.634600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.634582 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b165948d-4b0c-4df0-b225-3675562ae7e3-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.634677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.634605 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljkzc\" (UniqueName: \"kubernetes.io/projected/be04bddc-7158-41f5-9556-cef105850742-kube-api-access-ljkzc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.634677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.634620 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be04bddc-7158-41f5-9556-cef105850742-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:30.636270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.636238 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b165948d-4b0c-4df0-b225-3675562ae7e3-kube-api-access-lvvsn" pod "b165948d-4b0c-4df0-b225-3675562ae7e3" (UID: "b165948d-4b0c-4df0-b225-3675562ae7e3"). InnerVolumeSpecName "kube-api-access-lvvsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:30.736088 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:30.736047 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvvsn\" (UniqueName: \"kubernetes.io/projected/b165948d-4b0c-4df0-b225-3675562ae7e3-kube-api-access-lvvsn\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:31.193423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.193370 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" podUID="be04bddc-7158-41f5-9556-cef105850742" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.202:50052\" within 1s: context deadline exceeded" Nov 03 19:15:31.393197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.393165 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" Nov 03 19:15:31.393197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.393181 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs" event={"ID":"b165948d-4b0c-4df0-b225-3675562ae7e3","Type":"ContainerDied","Data":"2adc0fc023263b5e2abd861506efeb1d64b071b6cefc7658207faed5218227d2"} Nov 03 19:15:31.393457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.393277 4948 scope.go:122] "RemoveContainer" containerID="f3f3086442e2725d60e024b0a7f1c83aeddcc3ffe71a505b5057a54ab0575659" Nov 03 19:15:31.394957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.394932 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:15:31.395044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.394982 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:31.395103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.395084 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" event={"ID":"be04bddc-7158-41f5-9556-cef105850742","Type":"ContainerDied","Data":"fb87cbfae13539baba33edaf63961332ffc0e09946f6d3f8b4aae43727dce44a"} Nov 03 19:15:31.395188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.395162 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62" Nov 03 19:15:31.397193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.397164 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" event={"ID":"39803e8e-ba6f-449a-9d53-116d0147e05d","Type":"ContainerDied","Data":"a7d93fc2b4ee08963a8647fadd57ae282c78a824414c68db622303db13a6530b"} Nov 03 19:15:31.397193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.397189 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5" Nov 03 19:15:31.406986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.406964 4948 scope.go:122] "RemoveContainer" containerID="dafec5197639b65495ca8012285aef277fbe1d84a5e4c71a2750aac8fc9c66ed" Nov 03 19:15:31.417312 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.417288 4948 scope.go:122] "RemoveContainer" containerID="ea555dd5988dfa441f6828aa68093fc36eecdc00030d2c6e2524db91dfe5a12a" Nov 03 19:15:31.634975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.634901 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5"] Nov 03 19:15:31.639096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.639054 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecf40a-openshift-default-5fc9fb679c-7fhf5"] Nov 03 19:15:31.750300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.750265 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs"] Nov 03 19:15:31.756999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.756968 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e1f8f8-9d858999-6gpzs"] Nov 03 19:15:31.866342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.866300 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62"] Nov 03 19:15:31.870799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:31.870770 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e1f8f8-67d6f99995-blb62"] Nov 03 19:15:32.755578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:32.755531 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" path="/var/lib/kubelet/pods/39803e8e-ba6f-449a-9d53-116d0147e05d/volumes" Nov 03 19:15:32.756066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:32.756049 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b165948d-4b0c-4df0-b225-3675562ae7e3" path="/var/lib/kubelet/pods/b165948d-4b0c-4df0-b225-3675562ae7e3/volumes" Nov 03 19:15:32.756447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:32.756434 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be04bddc-7158-41f5-9556-cef105850742" path="/var/lib/kubelet/pods/be04bddc-7158-41f5-9556-cef105850742/volumes" Nov 03 19:15:32.797609 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:32.797574 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:15:38.642032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:38.641992 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r"] Nov 03 19:15:38.761309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:38.761276 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw"] Nov 03 19:15:43.642774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:43.642682 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" podUID="10cf7280-59ea-4c15-b677-de5cf836f300" containerName="payload-processing" containerID="cri-o://34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f" gracePeriod=25 Nov 03 19:15:43.762552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:43.762505 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" podUID="c3e59596-cda8-46bd-a896-4dd507707ec9" containerName="payload-pre-processing" containerID="cri-o://6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45" gracePeriod=25 Nov 03 19:15:43.943589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:43.943558 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:15:44.006502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.006463 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cf7280-59ea-4c15-b677-de5cf836f300-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cf7280-59ea-4c15-b677-de5cf836f300-plugins-config-volume\") pod \"10cf7280-59ea-4c15-b677-de5cf836f300\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " Nov 03 19:15:44.006728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.006527 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cf7280-59ea-4c15-b677-de5cf836f300-kube-api-access-nc6rh\" (UniqueName: \"kubernetes.io/projected/10cf7280-59ea-4c15-b677-de5cf836f300-kube-api-access-nc6rh\") pod \"10cf7280-59ea-4c15-b677-de5cf836f300\" (UID: \"10cf7280-59ea-4c15-b677-de5cf836f300\") " Nov 03 19:15:44.007928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.007486 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cf7280-59ea-4c15-b677-de5cf836f300-plugins-config-volume" pod "10cf7280-59ea-4c15-b677-de5cf836f300" (UID: "10cf7280-59ea-4c15-b677-de5cf836f300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:44.012981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.012956 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cf7280-59ea-4c15-b677-de5cf836f300-kube-api-access-nc6rh" pod "10cf7280-59ea-4c15-b677-de5cf836f300" (UID: "10cf7280-59ea-4c15-b677-de5cf836f300"). InnerVolumeSpecName "kube-api-access-nc6rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:44.059812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.059784 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:15:44.107876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.107831 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e59596-cda8-46bd-a896-4dd507707ec9-kube-api-access-sdbch\" (UniqueName: \"kubernetes.io/projected/c3e59596-cda8-46bd-a896-4dd507707ec9-kube-api-access-sdbch\") pod \"c3e59596-cda8-46bd-a896-4dd507707ec9\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " Nov 03 19:15:44.108104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.107934 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e59596-cda8-46bd-a896-4dd507707ec9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e59596-cda8-46bd-a896-4dd507707ec9-plugins-config-volume\") pod \"c3e59596-cda8-46bd-a896-4dd507707ec9\" (UID: \"c3e59596-cda8-46bd-a896-4dd507707ec9\") " Nov 03 19:15:44.108674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.108648 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cf7280-59ea-4c15-b677-de5cf836f300-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:44.108796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.108677 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc6rh\" (UniqueName: \"kubernetes.io/projected/10cf7280-59ea-4c15-b677-de5cf836f300-kube-api-access-nc6rh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:44.108796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.108729 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e59596-cda8-46bd-a896-4dd507707ec9-plugins-config-volume" pod "c3e59596-cda8-46bd-a896-4dd507707ec9" (UID: "c3e59596-cda8-46bd-a896-4dd507707ec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:44.111146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.111108 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e59596-cda8-46bd-a896-4dd507707ec9-kube-api-access-sdbch" pod "c3e59596-cda8-46bd-a896-4dd507707ec9" (UID: "c3e59596-cda8-46bd-a896-4dd507707ec9"). InnerVolumeSpecName "kube-api-access-sdbch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:44.210091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.210052 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbch\" (UniqueName: \"kubernetes.io/projected/c3e59596-cda8-46bd-a896-4dd507707ec9-kube-api-access-sdbch\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:44.210091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.210094 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e59596-cda8-46bd-a896-4dd507707ec9-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:44.566339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.566260 4948 generic.go:356] "Generic (PLEG): container finished" podID="10cf7280-59ea-4c15-b677-de5cf836f300" containerID="34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f" exitCode=0 Nov 03 19:15:44.566339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.566324 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" Nov 03 19:15:44.566551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.566337 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" event={"ID":"10cf7280-59ea-4c15-b677-de5cf836f300","Type":"ContainerDied","Data":"34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f"} Nov 03 19:15:44.566551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.566423 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r" event={"ID":"10cf7280-59ea-4c15-b677-de5cf836f300","Type":"ContainerDied","Data":"055101c2b16d20bfabed0535678f7e724ac13f4214ed40b22217c5330b1a3363"} Nov 03 19:15:44.566551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.566452 4948 scope.go:122] "RemoveContainer" containerID="34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f" Nov 03 19:15:44.568350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.568320 4948 generic.go:356] "Generic (PLEG): container finished" podID="c3e59596-cda8-46bd-a896-4dd507707ec9" containerID="6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45" exitCode=0 Nov 03 19:15:44.568475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.568420 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" event={"ID":"c3e59596-cda8-46bd-a896-4dd507707ec9","Type":"ContainerDied","Data":"6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45"} Nov 03 19:15:44.568550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.568505 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" event={"ID":"c3e59596-cda8-46bd-a896-4dd507707ec9","Type":"ContainerDied","Data":"9b01360eb8deebc3da2e9e53ebfec5bd7281d59363276c85348fe76460f5c32b"} Nov 03 19:15:44.568647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.568626 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw" Nov 03 19:15:44.578465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.578448 4948 scope.go:122] "RemoveContainer" containerID="34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f" Nov 03 19:15:44.578813 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:44.578788 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f\": container with ID starting with 34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f not found: ID does not exist" containerID="34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f" Nov 03 19:15:44.578899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.578827 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f"} err="failed to get container status \"34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f\": rpc error: code = NotFound desc = could not find container \"34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f\": container with ID starting with 34603bb2e1f6f0913f5cbb8c60964fe1925638994fb07b1f5ae2a1c1615d091f not found: ID does not exist" Nov 03 19:15:44.578899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.578854 4948 scope.go:122] "RemoveContainer" containerID="6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45" Nov 03 19:15:44.589492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.589476 4948 scope.go:122] "RemoveContainer" containerID="6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45" Nov 03 19:15:44.589802 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:15:44.589782 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45\": container with ID starting with 6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45 not found: ID does not exist" containerID="6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45" Nov 03 19:15:44.589868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.589816 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45"} err="failed to get container status \"6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45\": rpc error: code = NotFound desc = could not find container \"6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45\": container with ID starting with 6db49fabd4adb36758cf0a9f0168b16208481da7792bacdbec93709524074f45 not found: ID does not exist" Nov 03 19:15:44.738555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.738514 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw"] Nov 03 19:15:44.789237 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.789201 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a93912-574cd97df5-w5gbw"] Nov 03 19:15:44.853691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.853607 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r"] Nov 03 19:15:44.859145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:44.859119 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a93912-6dc7f84989-48g7r"] Nov 03 19:15:46.756472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:46.756434 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cf7280-59ea-4c15-b677-de5cf836f300" path="/var/lib/kubelet/pods/10cf7280-59ea-4c15-b677-de5cf836f300/volumes" Nov 03 19:15:46.756957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:46.756914 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e59596-cda8-46bd-a896-4dd507707ec9" path="/var/lib/kubelet/pods/c3e59596-cda8-46bd-a896-4dd507707ec9/volumes" Nov 03 19:15:59.926824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:15:59.926773 4948 scope.go:122] "RemoveContainer" containerID="6050965e63244d0f0b88b05a525a18a51c6e00e71fcb7d0fd9d86b3c10496ce8" Nov 03 19:16:02.797418 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:16:02.797314 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:16:19.009788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.009754 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr"] Nov 03 19:16:19.010457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010441 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010457 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010472 4948 cpu_manager.go:414] "containerMap: removing container" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010477 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010484 4948 cpu_manager.go:414] "containerMap: removing container" podUID="186aeaf8-9f70-416a-b468-8afc641cfaa4" containerName="revoke-keys" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010489 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186aeaf8-9f70-416a-b468-8afc641cfaa4" containerName="revoke-keys" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010506 4948 cpu_manager.go:414] "containerMap: removing container" podUID="10cf7280-59ea-4c15-b677-de5cf836f300" containerName="payload-processing" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010511 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cf7280-59ea-4c15-b677-de5cf836f300" containerName="payload-processing" Nov 03 19:16:19.010503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010518 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010523 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010537 4948 cpu_manager.go:414] "containerMap: removing container" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010541 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961a3275-52b1-443b-adcc-43af86bddcef" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010561 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b165948d-4b0c-4df0-b225-3675562ae7e3" containerName="payload-pre-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010566 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b165948d-4b0c-4df0-b225-3675562ae7e3" containerName="payload-pre-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010577 4948 cpu_manager.go:414] "containerMap: removing container" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010581 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010586 4948 cpu_manager.go:414] "containerMap: removing container" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010592 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010599 4948 cpu_manager.go:414] "containerMap: removing container" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010604 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32555649-8e98-4fb1-a9a6-29b5a9d6b90e" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010617 4948 cpu_manager.go:414] "containerMap: removing container" podUID="be04bddc-7158-41f5-9556-cef105850742" containerName="payload-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010621 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be04bddc-7158-41f5-9556-cef105850742" containerName="payload-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010631 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c3e59596-cda8-46bd-a896-4dd507707ec9" containerName="payload-pre-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010636 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e59596-cda8-46bd-a896-4dd507707ec9" containerName="payload-pre-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010643 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010647 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb3eff7-234f-4b79-9e31-69e75efc0fce" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010766 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010777 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b165948d-4b0c-4df0-b225-3675562ae7e3" containerName="payload-pre-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010783 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb51c78-b827-447d-9603-8d18eb55fed1" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010789 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="510004c0-3875-4da4-af50-af656c67668f" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010796 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77fb6c3-2ae1-4eed-8883-d39ddc4da728" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010801 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="be04bddc-7158-41f5-9556-cef105850742" containerName="payload-processing" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010809 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb756b1-a9d3-4235-980d-d9009a6226ac" containerName="cleanup" Nov 03 19:16:19.010807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010815 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e59596-cda8-46bd-a896-4dd507707ec9" containerName="payload-pre-processing" Nov 03 19:16:19.011588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010825 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="186aeaf8-9f70-416a-b468-8afc641cfaa4" containerName="revoke-keys" Nov 03 19:16:19.011588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.010833 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cf7280-59ea-4c15-b677-de5cf836f300" containerName="payload-processing" Nov 03 19:16:19.014576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.014558 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.076770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.071631 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr"] Nov 03 19:16:19.114828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.114790 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9dd9f2-serving-cert\"" Nov 03 19:16:19.164180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.164132 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-metrics-certs\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.164358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.164190 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-maas-api-tls\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.164358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.164297 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c04ffd-044b-4499-ac87-71d9fd5d238a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.164452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.164357 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqcn\" (UniqueName: \"kubernetes.io/projected/47c04ffd-044b-4499-ac87-71d9fd5d238a-kube-api-access-mxqcn\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.266164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.266078 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-metrics-certs\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.266164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.266126 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-maas-api-tls\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.266385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.266181 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c04ffd-044b-4499-ac87-71d9fd5d238a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.266385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.266239 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqcn\" (UniqueName: \"kubernetes.io/projected/47c04ffd-044b-4499-ac87-71d9fd5d238a-kube-api-access-mxqcn\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.267351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.267315 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c04ffd-044b-4499-ac87-71d9fd5d238a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.268895 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.268867 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-metrics-certs\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.268989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.268893 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-maas-api-tls\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.329088 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.329055 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqcn\" (UniqueName: \"kubernetes.io/projected/47c04ffd-044b-4499-ac87-71d9fd5d238a-kube-api-access-mxqcn\") pod \"maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:19.628809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:19.628731 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:20.022862 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:16:20.022824 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c04ffd_044b_4499_ac87_71d9fd5d238a.slice/crio-3d6f6bb7fbc93af49a3289f7bd288f458cd0b2407365478a476300d8014f9e1d WatchSource:0}: Error finding container 3d6f6bb7fbc93af49a3289f7bd288f458cd0b2407365478a476300d8014f9e1d: Status 404 returned error can't find the container with id 3d6f6bb7fbc93af49a3289f7bd288f458cd0b2407365478a476300d8014f9e1d Nov 03 19:16:20.023976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:20.023952 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr"] Nov 03 19:16:20.024761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:20.024743 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:16:20.957806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:20.957773 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" event={"ID":"47c04ffd-044b-4499-ac87-71d9fd5d238a","Type":"ContainerStarted","Data":"3d6f6bb7fbc93af49a3289f7bd288f458cd0b2407365478a476300d8014f9e1d"} Nov 03 19:16:22.021573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.021522 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" event={"ID":"47c04ffd-044b-4499-ac87-71d9fd5d238a","Type":"ContainerStarted","Data":"35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7"} Nov 03 19:16:22.022148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.021974 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:22.142082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.142030 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" podStartSLOduration=3.244827597 podStartE2EDuration="4.142015062s" totalImagesPullingTime="897.187465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:22.141787911 +0000 UTC m=+2311.624783824" watchObservedRunningTime="2026-11-03 19:16:22.142015062 +0000 UTC m=+2311.625010977" Nov 03 19:16:22.520568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.520530 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln"] Nov 03 19:16:22.520815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.520773 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" containerID="cri-o://1ea0376aa361e4de421157bfc176ab31e6d49442cdea9f3f47366b8f2053aa7f" gracePeriod=30 Nov 03 19:16:22.570286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.570250 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c"] Nov 03 19:16:22.570512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.570448 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" containerID="cri-o://ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90" gracePeriod=30 Nov 03 19:16:22.575261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.575232 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm"] Nov 03 19:16:22.575474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.575436 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" containerID="cri-o://0eaedeb657271febffca14817fdf9134339da1f3a407a2e77f5db97a5f76b2af" gracePeriod=30 Nov 03 19:16:22.579411 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.579387 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc"] Nov 03 19:16:22.579575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:22.579555 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" containerID="cri-o://c267e66b4781e6ebc33bb4eadf8bf62afd4fa6275769fe92c3eedcce731b0981" gracePeriod=30 Nov 03 19:16:27.294827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:27.294783 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-r6cjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:16:27.295226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:27.294858 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:16:27.605433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:27.605397 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-c62ln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 19:16:27.605582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:27.605456 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 19:16:27.782588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:27.782542 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv"] Nov 03 19:16:27.783202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:27.783172 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" podUID="a286ad3e-d341-42bb-a304-d92156e43b02" containerName="maas-api" containerID="cri-o://7add863121078f8e74ab586c8c334c04f4a52eb26e65a57c2a9be76e2a4abf2e" gracePeriod=30 Nov 03 19:16:27.936789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:27.936065 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006080 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-certs\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006271 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-credential-socket\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006350 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3d026d-b159-4d44-84ef-a20554d43c80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d026d-b159-4d44-84ef-a20554d43c80-istiod-ca-cert\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006408 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-data\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006472 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-token\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006547 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-kube-api-access-jr4dt\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-kube-api-access-jr4dt\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006619 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-envoy\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006658 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-podinfo\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.006765 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-socket\") pod \"9a3d026d-b159-4d44-84ef-a20554d43c80\" (UID: \"9a3d026d-b159-4d44-84ef-a20554d43c80\") " Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.007670 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-socket" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.007793 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-data" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.007956 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3d026d-b159-4d44-84ef-a20554d43c80-istiod-ca-cert" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.008018 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-credential-socket" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.008128 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-certs" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.036722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.023359 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-envoy" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.094361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.094251 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-kube-api-access-jr4dt" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "kube-api-access-jr4dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.104615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.099262 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:16:28.104615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.102542 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-token" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.106468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.106434 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-podinfo" pod "9a3d026d-b159-4d44-84ef-a20554d43c80" (UID: "9a3d026d-b159-4d44-84ef-a20554d43c80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:28.113665 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113633 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113670 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d026d-b159-4d44-84ef-a20554d43c80-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113686 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113717 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113733 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr4dt\" (UniqueName: \"kubernetes.io/projected/9a3d026d-b159-4d44-84ef-a20554d43c80-kube-api-access-jr4dt\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113747 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113761 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d026d-b159-4d44-84ef-a20554d43c80-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113775 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.113812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113789 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d026d-b159-4d44-84ef-a20554d43c80-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.114224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.113938 4948 generic.go:356] "Generic (PLEG): container finished" podID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerID="0eaedeb657271febffca14817fdf9134339da1f3a407a2e77f5db97a5f76b2af" exitCode=0 Nov 03 19:16:28.114224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.114089 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" event={"ID":"e19168cf-c7fb-4d33-a35d-32c3ea6cc372","Type":"ContainerDied","Data":"0eaedeb657271febffca14817fdf9134339da1f3a407a2e77f5db97a5f76b2af"} Nov 03 19:16:28.116763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.116719 4948 generic.go:356] "Generic (PLEG): container finished" podID="a286ad3e-d341-42bb-a304-d92156e43b02" containerID="7add863121078f8e74ab586c8c334c04f4a52eb26e65a57c2a9be76e2a4abf2e" exitCode=0 Nov 03 19:16:28.116881 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.116861 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" event={"ID":"a286ad3e-d341-42bb-a304-d92156e43b02","Type":"ContainerDied","Data":"7add863121078f8e74ab586c8c334c04f4a52eb26e65a57c2a9be76e2a4abf2e"} Nov 03 19:16:28.119123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.119103 4948 generic.go:356] "Generic (PLEG): container finished" podID="9f18b1dc-f307-4945-800f-0af6509176f5" containerID="1ea0376aa361e4de421157bfc176ab31e6d49442cdea9f3f47366b8f2053aa7f" exitCode=0 Nov 03 19:16:28.119227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.119195 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" event={"ID":"9f18b1dc-f307-4945-800f-0af6509176f5","Type":"ContainerDied","Data":"1ea0376aa361e4de421157bfc176ab31e6d49442cdea9f3f47366b8f2053aa7f"} Nov 03 19:16:28.123888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.123175 4948 generic.go:356] "Generic (PLEG): container finished" podID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerID="c267e66b4781e6ebc33bb4eadf8bf62afd4fa6275769fe92c3eedcce731b0981" exitCode=0 Nov 03 19:16:28.123888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.123269 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" event={"ID":"f781701e-aa01-4167-84f2-0e2c7a7601dc","Type":"ContainerDied","Data":"c267e66b4781e6ebc33bb4eadf8bf62afd4fa6275769fe92c3eedcce731b0981"} Nov 03 19:16:28.123888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.123320 4948 scope.go:122] "RemoveContainer" containerID="4a22ef7fd92ba9b0a12c40408a99ebce65cc489bc9ebf24d8d43afbd65ad3c5b" Nov 03 19:16:28.126403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.126381 4948 generic.go:356] "Generic (PLEG): container finished" podID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerID="ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90" exitCode=0 Nov 03 19:16:28.126579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.126554 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" event={"ID":"9a3d026d-b159-4d44-84ef-a20554d43c80","Type":"ContainerDied","Data":"ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90"} Nov 03 19:16:28.126659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.126607 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" event={"ID":"9a3d026d-b159-4d44-84ef-a20554d43c80","Type":"ContainerDied","Data":"ddfccda635e28d1b384e1489a8c64e2e5dd705d52af857dafbfad945a39eace0"} Nov 03 19:16:28.126823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.126712 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c" Nov 03 19:16:28.132823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.132802 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:16:28.140543 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.138795 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 19:16:28.143195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.142766 4948 scope.go:122] "RemoveContainer" containerID="ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90" Nov 03 19:16:28.146735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.146420 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:16:28.155166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.155143 4948 scope.go:122] "RemoveContainer" containerID="ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90" Nov 03 19:16:28.155588 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:16:28.155566 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90\": container with ID starting with ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90 not found: ID does not exist" containerID="ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90" Nov 03 19:16:28.155709 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.155599 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90"} err="failed to get container status \"ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90\": rpc error: code = NotFound desc = could not find container \"ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90\": container with ID starting with ca1970b0934bfd38d59932988b109ad832d430968192f21197c81c8224971c90 not found: ID does not exist" Nov 03 19:16:28.214964 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.214930 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-certs\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.215169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.214972 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-kube-api-access-tpr96\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-kube-api-access-tpr96\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215001 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-kube-api-access-lbdw8\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-kube-api-access-lbdw8\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215044 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-data\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215097 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-credential-socket\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215128 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-certs\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215164 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-socket\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215196 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-podinfo\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215238 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-token\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-token\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215269 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-certs" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215289 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-token\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-token\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215317 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-socket\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215406 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-data\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215471 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-credential-socket\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215507 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f18b1dc-f307-4945-800f-0af6509176f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f18b1dc-f307-4945-800f-0af6509176f5-istio-podinfo\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215552 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-istio-token\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-istio-token\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215644 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-envoy\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215682 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f18b1dc-f307-4945-800f-0af6509176f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f18b1dc-f307-4945-800f-0af6509176f5-istiod-ca-cert\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215732 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istiod-ca-cert\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215786 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-data\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215829 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-socket\") pod \"9f18b1dc-f307-4945-800f-0af6509176f5\" (UID: \"9f18b1dc-f307-4945-800f-0af6509176f5\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215871 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f781701e-aa01-4167-84f2-0e2c7a7601dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781701e-aa01-4167-84f2-0e2c7a7601dc-istiod-ca-cert\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215909 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-certs\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.215974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215950 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-envoy\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.217667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.216007 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-podinfo\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.217667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.216043 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-kube-api-access-rj48s\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-kube-api-access-rj48s\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.217667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.216075 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-envoy\") pod \"f781701e-aa01-4167-84f2-0e2c7a7601dc\" (UID: \"f781701e-aa01-4167-84f2-0e2c7a7601dc\") " Nov 03 19:16:28.217667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.216115 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-credential-socket\") pod \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\" (UID: \"e19168cf-c7fb-4d33-a35d-32c3ea6cc372\") " Nov 03 19:16:28.217667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.217480 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.215497 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-credential-socket" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.217270 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-data" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.218063 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-socket" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.218251 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-certs" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.218526 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-data" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.218908 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f18b1dc-f307-4945-800f-0af6509176f5-istiod-ca-cert" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.219007 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istiod-ca-cert" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.219121 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-credential-socket" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.219293 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-data" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.219313 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-socket" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.219539 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-certs" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.219554 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f781701e-aa01-4167-84f2-0e2c7a7601dc-istiod-ca-cert" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.219796 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-socket" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.225210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.223688 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-credential-socket" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.229402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.229362 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-kube-api-access-rj48s" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "kube-api-access-rj48s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.230266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.230237 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f18b1dc-f307-4945-800f-0af6509176f5-istio-podinfo" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:28.232099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232049 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-envoy" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.232212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232095 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-podinfo" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:28.232212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232141 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-envoy" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.232212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232154 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-token" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.232380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232248 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-envoy" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:28.232433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232371 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-kube-api-access-lbdw8" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "kube-api-access-lbdw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.232433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232397 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-token" pod "f781701e-aa01-4167-84f2-0e2c7a7601dc" (UID: "f781701e-aa01-4167-84f2-0e2c7a7601dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.232543 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.232463 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-kube-api-access-tpr96" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "kube-api-access-tpr96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.233776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.233744 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-istio-token" pod "9f18b1dc-f307-4945-800f-0af6509176f5" (UID: "9f18b1dc-f307-4945-800f-0af6509176f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.237123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.237084 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-podinfo" pod "e19168cf-c7fb-4d33-a35d-32c3ea6cc372" (UID: "e19168cf-c7fb-4d33-a35d-32c3ea6cc372"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:28.272124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.272092 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:16:28.318819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.318790 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a286ad3e-d341-42bb-a304-d92156e43b02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a286ad3e-d341-42bb-a304-d92156e43b02-openshift-service-ca\") pod \"a286ad3e-d341-42bb-a304-d92156e43b02\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " Nov 03 19:16:28.319287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.318918 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-metrics-certs\") pod \"a286ad3e-d341-42bb-a304-d92156e43b02\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " Nov 03 19:16:28.319287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319000 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a286ad3e-d341-42bb-a304-d92156e43b02-kube-api-access-5tkrl\" (UniqueName: \"kubernetes.io/projected/a286ad3e-d341-42bb-a304-d92156e43b02-kube-api-access-5tkrl\") pod \"a286ad3e-d341-42bb-a304-d92156e43b02\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " Nov 03 19:16:28.319287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319033 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls\") pod \"a286ad3e-d341-42bb-a304-d92156e43b02\" (UID: \"a286ad3e-d341-42bb-a304-d92156e43b02\") " Nov 03 19:16:28.319287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319126 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a286ad3e-d341-42bb-a304-d92156e43b02-openshift-service-ca" pod "a286ad3e-d341-42bb-a304-d92156e43b02" (UID: "a286ad3e-d341-42bb-a304-d92156e43b02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:28.319597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319585 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319602 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319616 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319629 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319642 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319655 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319671 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319680 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319689 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319715 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f18b1dc-f307-4945-800f-0af6509176f5-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319725 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319734 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a286ad3e-d341-42bb-a304-d92156e43b02-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319743 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319751 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f18b1dc-f307-4945-800f-0af6509176f5-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319759 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319767 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319775 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f18b1dc-f307-4945-800f-0af6509176f5-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319785 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781701e-aa01-4167-84f2-0e2c7a7601dc-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319797 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319805 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319814 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319823 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj48s\" (UniqueName: \"kubernetes.io/projected/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-kube-api-access-rj48s\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319834 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319843 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19168cf-c7fb-4d33-a35d-32c3ea6cc372-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319851 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpr96\" (UniqueName: \"kubernetes.io/projected/9f18b1dc-f307-4945-800f-0af6509176f5-kube-api-access-tpr96\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319863 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdw8\" (UniqueName: \"kubernetes.io/projected/f781701e-aa01-4167-84f2-0e2c7a7601dc-kube-api-access-lbdw8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.319927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.319877 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781701e-aa01-4167-84f2-0e2c7a7601dc-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.321168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.321073 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-metrics-certs" pod "a286ad3e-d341-42bb-a304-d92156e43b02" (UID: "a286ad3e-d341-42bb-a304-d92156e43b02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:28.321168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.321092 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls" pod "a286ad3e-d341-42bb-a304-d92156e43b02" (UID: "a286ad3e-d341-42bb-a304-d92156e43b02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:28.321462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.321442 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a286ad3e-d341-42bb-a304-d92156e43b02-kube-api-access-5tkrl" pod "a286ad3e-d341-42bb-a304-d92156e43b02" (UID: "a286ad3e-d341-42bb-a304-d92156e43b02"). InnerVolumeSpecName "kube-api-access-5tkrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:28.436793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.427969 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.436793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.428007 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tkrl\" (UniqueName: \"kubernetes.io/projected/a286ad3e-d341-42bb-a304-d92156e43b02-kube-api-access-5tkrl\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.436793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.428024 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a286ad3e-d341-42bb-a304-d92156e43b02-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:28.611916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.611884 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c"] Nov 03 19:16:28.624759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.624721 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4w4c"] Nov 03 19:16:28.763902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:28.763814 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" path="/var/lib/kubelet/pods/9a3d026d-b159-4d44-84ef-a20554d43c80/volumes" Nov 03 19:16:29.135759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.135635 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" Nov 03 19:16:29.135759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.135661 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm" event={"ID":"e19168cf-c7fb-4d33-a35d-32c3ea6cc372","Type":"ContainerDied","Data":"b451afb21ad14ec0f49c97106bcd3c8a9d521583d6eeaeeb80dcf93ea15b14c8"} Nov 03 19:16:29.135759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.135741 4948 scope.go:122] "RemoveContainer" containerID="0eaedeb657271febffca14817fdf9134339da1f3a407a2e77f5db97a5f76b2af" Nov 03 19:16:29.137232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.137215 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" Nov 03 19:16:29.137332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.137251 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv" event={"ID":"a286ad3e-d341-42bb-a304-d92156e43b02","Type":"ContainerDied","Data":"8ffd5fd5c61b6d8aa7dbb6697bc11378f90ffdb494da326b21d7acd265a08f20"} Nov 03 19:16:29.139042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.139021 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" Nov 03 19:16:29.139042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.139026 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln" event={"ID":"9f18b1dc-f307-4945-800f-0af6509176f5","Type":"ContainerDied","Data":"05bfe58fd64b8a978093908523fc492642be51babcfc982ff707d0310df28f33"} Nov 03 19:16:29.140754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.140737 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" Nov 03 19:16:29.140880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.140734 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc" event={"ID":"f781701e-aa01-4167-84f2-0e2c7a7601dc","Type":"ContainerDied","Data":"f8ad043bec05eaeb4ccd88456d9624dec0a61c1f13e98c44c3d8fd27a911bd9d"} Nov 03 19:16:29.148880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.148860 4948 scope.go:122] "RemoveContainer" containerID="7add863121078f8e74ab586c8c334c04f4a52eb26e65a57c2a9be76e2a4abf2e" Nov 03 19:16:29.159418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.159399 4948 scope.go:122] "RemoveContainer" containerID="1ea0376aa361e4de421157bfc176ab31e6d49442cdea9f3f47366b8f2053aa7f" Nov 03 19:16:29.172391 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.172374 4948 scope.go:122] "RemoveContainer" containerID="c267e66b4781e6ebc33bb4eadf8bf62afd4fa6275769fe92c3eedcce731b0981" Nov 03 19:16:29.310438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.310404 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln"] Nov 03 19:16:29.361200 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.361167 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-c62ln"] Nov 03 19:16:29.426082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.425992 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv"] Nov 03 19:16:29.429899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.429877 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-97e020-867fd58c7f-p5kjv"] Nov 03 19:16:29.542969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.542932 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm"] Nov 03 19:16:29.551743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.551710 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-r6cjm"] Nov 03 19:16:29.750001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.749958 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc"] Nov 03 19:16:29.777915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.777876 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vtbjc"] Nov 03 19:16:29.927628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.927564 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx"] Nov 03 19:16:29.931540 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:29.931509 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-97e020-n7xqx"] Nov 03 19:16:30.753997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:30.753961 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66babf83-5e76-4406-8c0b-a8a30ccbfe1c" path="/var/lib/kubelet/pods/66babf83-5e76-4406-8c0b-a8a30ccbfe1c/volumes" Nov 03 19:16:30.754478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:30.754464 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" path="/var/lib/kubelet/pods/9f18b1dc-f307-4945-800f-0af6509176f5/volumes" Nov 03 19:16:30.754928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:30.754913 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a286ad3e-d341-42bb-a304-d92156e43b02" path="/var/lib/kubelet/pods/a286ad3e-d341-42bb-a304-d92156e43b02/volumes" Nov 03 19:16:30.755666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:30.755651 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" path="/var/lib/kubelet/pods/e19168cf-c7fb-4d33-a35d-32c3ea6cc372/volumes" Nov 03 19:16:30.756110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:30.756097 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" path="/var/lib/kubelet/pods/f781701e-aa01-4167-84f2-0e2c7a7601dc/volumes" Nov 03 19:16:31.888930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.888881 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn"] Nov 03 19:16:31.889876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889855 4948 cpu_manager.go:414] "containerMap: removing container" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" Nov 03 19:16:31.889876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889876 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889897 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889904 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889942 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889950 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889962 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a286ad3e-d341-42bb-a304-d92156e43b02" containerName="maas-api" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889969 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a286ad3e-d341-42bb-a304-d92156e43b02" containerName="maas-api" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889985 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.889993 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890015 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890022 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890033 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" Nov 03 19:16:31.890053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890039 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" Nov 03 19:16:31.890687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890165 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f18b1dc-f307-4945-800f-0af6509176f5" containerName="istio-proxy" Nov 03 19:16:31.890687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890175 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" Nov 03 19:16:31.890687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890192 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3d026d-b159-4d44-84ef-a20554d43c80" containerName="istio-proxy" Nov 03 19:16:31.890687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890205 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="39803e8e-ba6f-449a-9d53-116d0147e05d" containerName="istio-proxy" Nov 03 19:16:31.890687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890216 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a286ad3e-d341-42bb-a304-d92156e43b02" containerName="maas-api" Nov 03 19:16:31.890687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.890227 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19168cf-c7fb-4d33-a35d-32c3ea6cc372" containerName="istio-proxy" Nov 03 19:16:31.896575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.896554 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:31.950884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.950846 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn"] Nov 03 19:16:31.996084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.996051 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:16:31.996260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:31.996146 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:16:32.004905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.004878 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7d5\" (UniqueName: \"kubernetes.io/projected/54703451-d2e7-45b6-9040-a6f86940e78a-kube-api-access-kw7d5\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.005014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.004998 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.005109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.005090 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.106729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.106677 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7d5\" (UniqueName: \"kubernetes.io/projected/54703451-d2e7-45b6-9040-a6f86940e78a-kube-api-access-kw7d5\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.106936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.106797 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.106936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.106869 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.107535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.107509 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.109327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.109301 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.170507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.170431 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7d5\" (UniqueName: \"kubernetes.io/projected/54703451-d2e7-45b6-9040-a6f86940e78a-kube-api-access-kw7d5\") pod \"maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.210892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.210853 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:32.593687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:32.593645 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn"] Nov 03 19:16:32.594773 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:16:32.594743 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54703451_d2e7_45b6_9040_a6f86940e78a.slice/crio-9afe689c4d7ebaa9b76201ac1399e4f12ef811bf34e3cc849fc7854cbf3cb6ac WatchSource:0}: Error finding container 9afe689c4d7ebaa9b76201ac1399e4f12ef811bf34e3cc849fc7854cbf3cb6ac: Status 404 returned error can't find the container with id 9afe689c4d7ebaa9b76201ac1399e4f12ef811bf34e3cc849fc7854cbf3cb6ac Nov 03 19:16:32.795550 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:16:32.795506 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:16:33.180303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:33.180269 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" event={"ID":"54703451-d2e7-45b6-9040-a6f86940e78a","Type":"ContainerStarted","Data":"e3bef5b59256c8a00b20106a605dee73f5f9d311701beaec5759e9c5ac580675"} Nov 03 19:16:33.180686 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:33.180334 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" event={"ID":"54703451-d2e7-45b6-9040-a6f86940e78a","Type":"ContainerStarted","Data":"9afe689c4d7ebaa9b76201ac1399e4f12ef811bf34e3cc849fc7854cbf3cb6ac"} Nov 03 19:16:33.311066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:33.310942 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" podStartSLOduration=2.31092334 podStartE2EDuration="2.31092334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:33.304955743 +0000 UTC m=+2322.787951660" watchObservedRunningTime="2026-11-03 19:16:33.31092334 +0000 UTC m=+2322.793919254" Nov 03 19:16:34.192647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:34.192613 4948 generic.go:356] "Generic (PLEG): container finished" podID="54703451-d2e7-45b6-9040-a6f86940e78a" containerID="e3bef5b59256c8a00b20106a605dee73f5f9d311701beaec5759e9c5ac580675" exitCode=0 Nov 03 19:16:34.193078 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:34.192721 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" event={"ID":"54703451-d2e7-45b6-9040-a6f86940e78a","Type":"ContainerDied","Data":"e3bef5b59256c8a00b20106a605dee73f5f9d311701beaec5759e9c5ac580675"} Nov 03 19:16:35.389450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.389423 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:35.455335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.455241 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-xks-ca\") pod \"54703451-d2e7-45b6-9040-a6f86940e78a\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " Nov 03 19:16:35.455335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.455298 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-openshift-ca\") pod \"54703451-d2e7-45b6-9040-a6f86940e78a\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " Nov 03 19:16:35.455579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.455532 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54703451-d2e7-45b6-9040-a6f86940e78a-kube-api-access-kw7d5\" (UniqueName: \"kubernetes.io/projected/54703451-d2e7-45b6-9040-a6f86940e78a-kube-api-access-kw7d5\") pod \"54703451-d2e7-45b6-9040-a6f86940e78a\" (UID: \"54703451-d2e7-45b6-9040-a6f86940e78a\") " Nov 03 19:16:35.455761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.455732 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-openshift-ca" pod "54703451-d2e7-45b6-9040-a6f86940e78a" (UID: "54703451-d2e7-45b6-9040-a6f86940e78a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:35.456136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.456109 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:35.457781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.457750 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54703451-d2e7-45b6-9040-a6f86940e78a-kube-api-access-kw7d5" pod "54703451-d2e7-45b6-9040-a6f86940e78a" (UID: "54703451-d2e7-45b6-9040-a6f86940e78a"). InnerVolumeSpecName "kube-api-access-kw7d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:35.457934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.457878 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-xks-ca" pod "54703451-d2e7-45b6-9040-a6f86940e78a" (UID: "54703451-d2e7-45b6-9040-a6f86940e78a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:35.558007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.557965 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54703451-d2e7-45b6-9040-a6f86940e78a-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:35.558007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:35.558005 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw7d5\" (UniqueName: \"kubernetes.io/projected/54703451-d2e7-45b6-9040-a6f86940e78a-kube-api-access-kw7d5\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:36.212612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:36.212580 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" Nov 03 19:16:36.212859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:36.212617 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn" event={"ID":"54703451-d2e7-45b6-9040-a6f86940e78a","Type":"ContainerDied","Data":"9afe689c4d7ebaa9b76201ac1399e4f12ef811bf34e3cc849fc7854cbf3cb6ac"} Nov 03 19:16:36.212859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:36.212673 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afe689c4d7ebaa9b76201ac1399e4f12ef811bf34e3cc849fc7854cbf3cb6ac" Nov 03 19:16:37.134809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:37.134774 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d"] Nov 03 19:16:37.135204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:37.134998 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" containerID="cri-o://73fb8443a70ed3117408bbb82babdda1bac22f02673ebd2ebcd0e0ffc197ea11" gracePeriod=30 Nov 03 19:16:38.864838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:38.864798 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428"] Nov 03 19:16:38.866400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:38.866371 4948 cpu_manager.go:414] "containerMap: removing container" podUID="54703451-d2e7-45b6-9040-a6f86940e78a" containerName="revoke-keys" Nov 03 19:16:38.866568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:38.866546 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54703451-d2e7-45b6-9040-a6f86940e78a" containerName="revoke-keys" Nov 03 19:16:38.866846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:38.866828 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="54703451-d2e7-45b6-9040-a6f86940e78a" containerName="revoke-keys" Nov 03 19:16:38.866975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:38.866961 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f781701e-aa01-4167-84f2-0e2c7a7601dc" containerName="istio-proxy" Nov 03 19:16:38.888338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:38.888304 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:38.963774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:38.952814 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3d74f801-openshift-default-dockercfg-dg7tq\"" Nov 03 19:16:39.038403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.038289 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42791c9a-d4ed-4055-b330-f565ef971dc7-istiod-ca-cert\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.054914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.054874 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-podinfo\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.068745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.055199 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-socket\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.357275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.353575 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-envoy\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.357275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.353673 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-token\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.357275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.353761 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-certs\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.357275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.353857 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-data\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.357275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.353890 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-credential-socket\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.357275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.353963 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprll\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-kube-api-access-vprll\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.357275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.261068 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428"] Nov 03 19:16:39.455656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.455620 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42791c9a-d4ed-4055-b330-f565ef971dc7-istiod-ca-cert\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.455902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.455717 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-podinfo\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.455902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.455770 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-socket\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.455902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.455800 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-envoy\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.455902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.455859 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-token\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.455909 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-certs\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.455990 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-data\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.456022 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-credential-socket\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.456087 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprll\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-kube-api-access-vprll\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.456205 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-socket\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.456462 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-certs\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.456505 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42791c9a-d4ed-4055-b330-f565ef971dc7-istiod-ca-cert\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.456520 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-data\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.456799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.456775 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-credential-socket\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.458531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.458497 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-envoy\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.459262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.459228 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-podinfo\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.522235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.522198 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-token\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.522418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.522275 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprll\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-kube-api-access-vprll\") pod \"e2e-mt-3d74f801-openshift-default-78ddf467c-kv428\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:39.777628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:39.777587 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:40.333543 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:16:40.333502 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42791c9a_d4ed_4055_b330_f565ef971dc7.slice/crio-e6831a8cd029673344e2ccdde2debc551d2229f1c10c64271c579fa2c87ceb10 WatchSource:0}: Error finding container e6831a8cd029673344e2ccdde2debc551d2229f1c10c64271c579fa2c87ceb10: Status 404 returned error can't find the container with id e6831a8cd029673344e2ccdde2debc551d2229f1c10c64271c579fa2c87ceb10 Nov 03 19:16:40.335715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:40.335672 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428"] Nov 03 19:16:40.336715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:40.336672 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:16:40.336799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:40.336785 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:16:40.336840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:40.336826 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:16:40.425216 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:40.425177 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" event={"ID":"42791c9a-d4ed-4055-b330-f565ef971dc7","Type":"ContainerStarted","Data":"e6831a8cd029673344e2ccdde2debc551d2229f1c10c64271c579fa2c87ceb10"} Nov 03 19:16:41.435471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:41.435431 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" event={"ID":"42791c9a-d4ed-4055-b330-f565ef971dc7","Type":"ContainerStarted","Data":"49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e"} Nov 03 19:16:41.666723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:41.666008 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" podStartSLOduration=3.665986507 podStartE2EDuration="3.665986507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:41.65981511 +0000 UTC m=+2331.142811028" watchObservedRunningTime="2026-11-03 19:16:41.665986507 +0000 UTC m=+2331.148982422" Nov 03 19:16:41.783784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:41.780036 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:41.812862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:41.809056 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:42.596952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.596916 4948 generic.go:356] "Generic (PLEG): container finished" podID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerID="73fb8443a70ed3117408bbb82babdda1bac22f02673ebd2ebcd0e0ffc197ea11" exitCode=0 Nov 03 19:16:42.597789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.597755 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" event={"ID":"5d0263c0-0903-4b42-94e5-2d65d5823087","Type":"ContainerDied","Data":"73fb8443a70ed3117408bbb82babdda1bac22f02673ebd2ebcd0e0ffc197ea11"} Nov 03 19:16:42.597896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.597863 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" event={"ID":"5d0263c0-0903-4b42-94e5-2d65d5823087","Type":"ContainerDied","Data":"23c029030dc107c3b959ff9af4ca92aa40b9cde3b695c5a10e74865ebb77c816"} Nov 03 19:16:42.597941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.597915 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c029030dc107c3b959ff9af4ca92aa40b9cde3b695c5a10e74865ebb77c816" Nov 03 19:16:42.597941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.597936 4948 scope.go:122] "RemoveContainer" containerID="a45a71dc16ec3ddf522bc0d2f8d16982d7c7945f602eb456b4edd527273cc3f5" Nov 03 19:16:42.598432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.598410 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:42.599489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.599474 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:16:42.614434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.614400 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714265 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-kube-api-access-px4jr\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-kube-api-access-px4jr\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714384 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0263c0-0903-4b42-94e5-2d65d5823087-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0263c0-0903-4b42-94e5-2d65d5823087-istiod-ca-cert\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714492 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-certs\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714601 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-envoy\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714639 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-token\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-token\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714677 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-socket\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714735 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-credential-socket\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714813 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-podinfo\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.715722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.714865 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-data\") pod \"5d0263c0-0903-4b42-94e5-2d65d5823087\" (UID: \"5d0263c0-0903-4b42-94e5-2d65d5823087\") " Nov 03 19:16:42.746806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.744231 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0263c0-0903-4b42-94e5-2d65d5823087-istiod-ca-cert" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:42.746806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.744684 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-certs" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.772869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.772829 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-socket" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.777267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.773296 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-data" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.804770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.802821 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-credential-socket" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.822744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.819556 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.822744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.819591 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.822744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.819607 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0263c0-0903-4b42-94e5-2d65d5823087-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.822744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.819622 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.822744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.819635 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.836305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.836256 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-kube-api-access-px4jr" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "kube-api-access-px4jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:42.856895 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.854455 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-podinfo" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:42.881720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.875890 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-token" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:42.881720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.875989 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-envoy" pod "5d0263c0-0903-4b42-94e5-2d65d5823087" (UID: "5d0263c0-0903-4b42-94e5-2d65d5823087"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.937195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.937098 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.937195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.937126 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.937195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.937136 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0263c0-0903-4b42-94e5-2d65d5823087-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.937195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.937146 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px4jr\" (UniqueName: \"kubernetes.io/projected/5d0263c0-0903-4b42-94e5-2d65d5823087-kube-api-access-px4jr\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.948148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.948109 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq"] Nov 03 19:16:42.954934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:42.954902 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a93912-p4wdq"] Nov 03 19:16:43.607660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:43.607630 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d" Nov 03 19:16:43.847034 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:43.846998 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d"] Nov 03 19:16:43.850842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:43.850813 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-rfb2d"] Nov 03 19:16:44.756099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:44.756064 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186aeaf8-9f70-416a-b468-8afc641cfaa4" path="/var/lib/kubelet/pods/186aeaf8-9f70-416a-b468-8afc641cfaa4/volumes" Nov 03 19:16:44.756631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:44.756613 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" path="/var/lib/kubelet/pods/5d0263c0-0903-4b42-94e5-2d65d5823087/volumes" Nov 03 19:16:48.378989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.378943 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2"] Nov 03 19:16:48.379731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.379713 4948 cpu_manager.go:414] "containerMap: removing container" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" Nov 03 19:16:48.379782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.379731 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" Nov 03 19:16:48.379782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.379769 4948 cpu_manager.go:414] "containerMap: removing container" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" Nov 03 19:16:48.379782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.379775 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" Nov 03 19:16:48.379891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.379876 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" Nov 03 19:16:48.379891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.379890 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0263c0-0903-4b42-94e5-2d65d5823087" containerName="istio-proxy" Nov 03 19:16:48.384616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.384597 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:48.491724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.491216 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2"] Nov 03 19:16:48.591753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.582918 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3d74f801-dockercfg-pc5bq\"" Nov 03 19:16:48.591753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.582993 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3d74f801\"" Nov 03 19:16:48.591753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.583242 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2qb\" (UniqueName: \"kubernetes.io/projected/2ce424f6-afbd-46af-af02-3a40594a498c-kube-api-access-dr2qb\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:48.591753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.583312 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce424f6-afbd-46af-af02-3a40594a498c-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:48.685001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.684964 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2qb\" (UniqueName: \"kubernetes.io/projected/2ce424f6-afbd-46af-af02-3a40594a498c-kube-api-access-dr2qb\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:48.685258 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.685042 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce424f6-afbd-46af-af02-3a40594a498c-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:48.685823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.685802 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce424f6-afbd-46af-af02-3a40594a498c-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:48.747578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:48.747542 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2qb\" (UniqueName: \"kubernetes.io/projected/2ce424f6-afbd-46af-af02-3a40594a498c-kube-api-access-dr2qb\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:49.001965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:49.001872 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:49.394644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:49.394615 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2"] Nov 03 19:16:49.395121 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:16:49.395096 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce424f6_afbd_46af_af02_3a40594a498c.slice/crio-396b18f3892077849090498827dd7a8a806dc260156683f1c6201c77fdf3edc6 WatchSource:0}: Error finding container 396b18f3892077849090498827dd7a8a806dc260156683f1c6201c77fdf3edc6: Status 404 returned error can't find the container with id 396b18f3892077849090498827dd7a8a806dc260156683f1c6201c77fdf3edc6 Nov 03 19:16:49.738938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:49.738894 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" event={"ID":"2ce424f6-afbd-46af-af02-3a40594a498c","Type":"ContainerStarted","Data":"180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63"} Nov 03 19:16:49.739250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:49.739225 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" event={"ID":"2ce424f6-afbd-46af-af02-3a40594a498c","Type":"ContainerStarted","Data":"396b18f3892077849090498827dd7a8a806dc260156683f1c6201c77fdf3edc6"} Nov 03 19:16:49.746513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:49.746481 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:16:49.858930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:49.858862 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" podStartSLOduration=1.8588407139999998 podStartE2EDuration="1.858840714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:49.857875152 +0000 UTC m=+2339.340871066" watchObservedRunningTime="2026-11-03 19:16:49.858840714 +0000 UTC m=+2339.341836629" Nov 03 19:16:50.876032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:50.876002 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg"] Nov 03 19:16:50.876438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:50.876190 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" containerID="cri-o://717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c" gracePeriod=30 Nov 03 19:16:52.083436 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.083396 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2"] Nov 03 19:16:52.083990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.083677 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" podUID="74a4c78c-c217-415f-82cc-683275be6bea" containerName="maas-api" containerID="cri-o://a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4" gracePeriod=30 Nov 03 19:16:52.414811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.414785 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:16:52.445576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.445547 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a4c78c-c217-415f-82cc-683275be6bea-kube-api-access-gkzss\" (UniqueName: \"kubernetes.io/projected/74a4c78c-c217-415f-82cc-683275be6bea-kube-api-access-gkzss\") pod \"74a4c78c-c217-415f-82cc-683275be6bea\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " Nov 03 19:16:52.445576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.445587 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-metrics-certs\") pod \"74a4c78c-c217-415f-82cc-683275be6bea\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " Nov 03 19:16:52.445830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.445623 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls\") pod \"74a4c78c-c217-415f-82cc-683275be6bea\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " Nov 03 19:16:52.445902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.445876 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a4c78c-c217-415f-82cc-683275be6bea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4c78c-c217-415f-82cc-683275be6bea-openshift-service-ca\") pod \"74a4c78c-c217-415f-82cc-683275be6bea\" (UID: \"74a4c78c-c217-415f-82cc-683275be6bea\") " Nov 03 19:16:52.446349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.446319 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4c78c-c217-415f-82cc-683275be6bea-openshift-service-ca" pod "74a4c78c-c217-415f-82cc-683275be6bea" (UID: "74a4c78c-c217-415f-82cc-683275be6bea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:52.446578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.446546 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4c78c-c217-415f-82cc-683275be6bea-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:52.447952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.447927 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls" pod "74a4c78c-c217-415f-82cc-683275be6bea" (UID: "74a4c78c-c217-415f-82cc-683275be6bea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:52.448055 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.447959 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a4c78c-c217-415f-82cc-683275be6bea-kube-api-access-gkzss" pod "74a4c78c-c217-415f-82cc-683275be6bea" (UID: "74a4c78c-c217-415f-82cc-683275be6bea"). InnerVolumeSpecName "kube-api-access-gkzss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:52.448055 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.447966 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-metrics-certs" pod "74a4c78c-c217-415f-82cc-683275be6bea" (UID: "74a4c78c-c217-415f-82cc-683275be6bea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:52.557811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.551139 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzss\" (UniqueName: \"kubernetes.io/projected/74a4c78c-c217-415f-82cc-683275be6bea-kube-api-access-gkzss\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:52.557811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.551188 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:52.557811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.551202 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a4c78c-c217-415f-82cc-683275be6bea-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:52.769066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.769020 4948 generic.go:356] "Generic (PLEG): container finished" podID="74a4c78c-c217-415f-82cc-683275be6bea" containerID="a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4" exitCode=0 Nov 03 19:16:52.769249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.769075 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" Nov 03 19:16:52.769249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.769103 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" event={"ID":"74a4c78c-c217-415f-82cc-683275be6bea","Type":"ContainerDied","Data":"a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4"} Nov 03 19:16:52.769249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.769165 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2" event={"ID":"74a4c78c-c217-415f-82cc-683275be6bea","Type":"ContainerDied","Data":"70ca036d025c02d95259d2641f492c8ac4aa63aa0ad3048ca306f0383f18c50e"} Nov 03 19:16:52.769249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.769191 4948 scope.go:122] "RemoveContainer" containerID="a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4" Nov 03 19:16:52.786310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.786287 4948 scope.go:122] "RemoveContainer" containerID="a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4" Nov 03 19:16:52.786688 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:16:52.786655 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4\": container with ID starting with a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4 not found: ID does not exist" containerID="a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4" Nov 03 19:16:52.786814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.786735 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4"} err="failed to get container status \"a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4\": rpc error: code = NotFound desc = could not find container \"a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4\": container with ID starting with a7d5279baa2994de3373bfe900999252673e06802387fb57218a970d983e6fc4 not found: ID does not exist" Nov 03 19:16:52.941402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.941370 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2"] Nov 03 19:16:52.945350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:52.945318 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba5674-5f8969588-8pxr2"] Nov 03 19:16:54.754609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:54.754577 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a4c78c-c217-415f-82cc-683275be6bea" path="/var/lib/kubelet/pods/74a4c78c-c217-415f-82cc-683275be6bea/volumes" Nov 03 19:16:55.401483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:55.401440 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:16:55.401667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:55.401500 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:16:56.181619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.181589 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:16:56.316674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316640 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-credential-socket\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.316894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316746 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-socket\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.316894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316810 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-istio-token\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-istio-token\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.316894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316852 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-kube-api-access-pcpj8\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-kube-api-access-pcpj8\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.317085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316897 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-certs\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.317085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316931 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc16153-2d17-4bb4-a11b-efb0355158be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc16153-2d17-4bb4-a11b-efb0355158be-istiod-ca-cert\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.317085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316978 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acc16153-2d17-4bb4-a11b-efb0355158be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc16153-2d17-4bb4-a11b-efb0355158be-istio-podinfo\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.317085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.316992 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-credential-socket" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:56.317372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.317086 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-envoy\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.317372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.317177 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-data\") pod \"acc16153-2d17-4bb4-a11b-efb0355158be\" (UID: \"acc16153-2d17-4bb4-a11b-efb0355158be\") " Nov 03 19:16:56.317686 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.317664 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.317953 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.317930 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-data" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:56.319654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.319630 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acc16153-2d17-4bb4-a11b-efb0355158be-istio-podinfo" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:56.320387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.320343 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-envoy" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:56.320644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.320607 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-certs" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:56.320644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.320628 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-socket" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:56.320850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.320666 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc16153-2d17-4bb4-a11b-efb0355158be-istiod-ca-cert" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:56.322538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.322501 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-kube-api-access-pcpj8" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "kube-api-access-pcpj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:56.323372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.323341 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-istio-token" pod "acc16153-2d17-4bb4-a11b-efb0355158be" (UID: "acc16153-2d17-4bb4-a11b-efb0355158be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.418930 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.418958 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.418967 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.418979 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.418992 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcpj8\" (UniqueName: \"kubernetes.io/projected/acc16153-2d17-4bb4-a11b-efb0355158be-kube-api-access-pcpj8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.419003 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc16153-2d17-4bb4-a11b-efb0355158be-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.419011 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc16153-2d17-4bb4-a11b-efb0355158be-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.419013 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.419019 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc16153-2d17-4bb4-a11b-efb0355158be-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:56.809322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.809282 4948 generic.go:356] "Generic (PLEG): container finished" podID="acc16153-2d17-4bb4-a11b-efb0355158be" containerID="717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c" exitCode=0 Nov 03 19:16:56.809502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.809357 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" Nov 03 19:16:56.809502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.809381 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" event={"ID":"acc16153-2d17-4bb4-a11b-efb0355158be","Type":"ContainerDied","Data":"717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c"} Nov 03 19:16:56.809502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.809446 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg" event={"ID":"acc16153-2d17-4bb4-a11b-efb0355158be","Type":"ContainerDied","Data":"dc44bfe2591d6a01105fe86a21ea591d15db3bf10df130fcae38d820bf81a922"} Nov 03 19:16:56.809502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.809477 4948 scope.go:122] "RemoveContainer" containerID="717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c" Nov 03 19:16:56.821102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.821077 4948 scope.go:122] "RemoveContainer" containerID="f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce" Nov 03 19:16:56.831867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.831831 4948 scope.go:122] "RemoveContainer" containerID="717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c" Nov 03 19:16:56.832244 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:16:56.832210 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c\": container with ID starting with 717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c not found: ID does not exist" containerID="717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c" Nov 03 19:16:56.832344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.832258 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c"} err="failed to get container status \"717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c\": rpc error: code = NotFound desc = could not find container \"717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c\": container with ID starting with 717fbe5b13b203838adf16794bf6dc939f0f48b55a9f056475a504c962ad137c not found: ID does not exist" Nov 03 19:16:56.832344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.832285 4948 scope.go:122] "RemoveContainer" containerID="f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce" Nov 03 19:16:56.832620 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:16:56.832590 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce\": container with ID starting with f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce not found: ID does not exist" containerID="f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce" Nov 03 19:16:56.832714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:56.832627 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce"} err="failed to get container status \"f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce\": rpc error: code = NotFound desc = could not find container \"f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce\": container with ID starting with f53001b6019f85b8c87bd542c6b26fffd4903d42ca18488f6c479b9033c087ce not found: ID does not exist" Nov 03 19:16:57.032343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:57.032241 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg"] Nov 03 19:16:57.085137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:57.085096 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a93912-openshift-default-65f75877d7-6pjbg"] Nov 03 19:16:58.756112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:16:58.756076 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" path="/var/lib/kubelet/pods/acc16153-2d17-4bb4-a11b-efb0355158be/volumes" Nov 03 19:17:00.076046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:00.076008 4948 scope.go:122] "RemoveContainer" containerID="04f8d30af86c9f3ba412a66fd491ee317e5da3d82640f46a38ecc0874ebec3d0" Nov 03 19:17:01.104670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.104632 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g"] Nov 03 19:17:01.105106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.104876 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" containerID="cri-o://f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49" gracePeriod=30 Nov 03 19:17:01.763902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.763871 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:17:01.850631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.850598 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts"] Nov 03 19:17:01.851300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851285 4948 cpu_manager.go:414] "containerMap: removing container" podUID="74a4c78c-c217-415f-82cc-683275be6bea" containerName="maas-api" Nov 03 19:17:01.851345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851302 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a4c78c-c217-415f-82cc-683275be6bea" containerName="maas-api" Nov 03 19:17:01.851345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851343 4948 cpu_manager.go:414] "containerMap: removing container" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" Nov 03 19:17:01.851418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851349 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" Nov 03 19:17:01.851418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851357 4948 cpu_manager.go:414] "containerMap: removing container" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" Nov 03 19:17:01.851418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851363 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" Nov 03 19:17:01.851507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851465 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" Nov 03 19:17:01.851507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851475 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc16153-2d17-4bb4-a11b-efb0355158be" containerName="istio-proxy" Nov 03 19:17:01.851507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.851485 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a4c78c-c217-415f-82cc-683275be6bea" containerName="maas-api" Nov 03 19:17:01.856067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.856043 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:01.912178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.912145 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:17:01.912416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.912395 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:17:01.915469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.915427 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts"] Nov 03 19:17:01.956538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.956491 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxcl\" (UniqueName: \"kubernetes.io/projected/95416322-89b0-4f41-b294-508f397deac2-kube-api-access-psxcl\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:01.956767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.956694 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95416322-89b0-4f41-b294-508f397deac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:01.957086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:01.957048 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95416322-89b0-4f41-b294-508f397deac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.058226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.058134 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95416322-89b0-4f41-b294-508f397deac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.058385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.058217 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95416322-89b0-4f41-b294-508f397deac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.058385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.058275 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxcl\" (UniqueName: \"kubernetes.io/projected/95416322-89b0-4f41-b294-508f397deac2-kube-api-access-psxcl\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.059128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.059092 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95416322-89b0-4f41-b294-508f397deac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.060841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.060814 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95416322-89b0-4f41-b294-508f397deac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.120341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.120311 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxcl\" (UniqueName: \"kubernetes.io/projected/95416322-89b0-4f41-b294-508f397deac2-kube-api-access-psxcl\") pod \"maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.171234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.171191 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:02.554328 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:17:02.554298 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95416322_89b0_4f41_b294_508f397deac2.slice/crio-61061f26b3695ecc5f7cbb9cb0817605122831abe3c38a49f8165a84498e9992 WatchSource:0}: Error finding container 61061f26b3695ecc5f7cbb9cb0817605122831abe3c38a49f8165a84498e9992: Status 404 returned error can't find the container with id 61061f26b3695ecc5f7cbb9cb0817605122831abe3c38a49f8165a84498e9992 Nov 03 19:17:02.554561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.554530 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts"] Nov 03 19:17:02.795138 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:02.795099 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:17:02.881778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.881745 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" event={"ID":"95416322-89b0-4f41-b294-508f397deac2","Type":"ContainerStarted","Data":"49f49af8127358b3fe13fef1c162f7da56a1d05236975a577782740cf23b608e"} Nov 03 19:17:02.881973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:02.881799 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" event={"ID":"95416322-89b0-4f41-b294-508f397deac2","Type":"ContainerStarted","Data":"61061f26b3695ecc5f7cbb9cb0817605122831abe3c38a49f8165a84498e9992"} Nov 03 19:17:03.001635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:03.001580 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" podStartSLOduration=2.001561642 podStartE2EDuration="2.001561642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:02.998897626 +0000 UTC m=+2352.481893539" watchObservedRunningTime="2026-11-03 19:17:03.001561642 +0000 UTC m=+2352.484557613" Nov 03 19:17:03.891746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:03.891641 4948 generic.go:356] "Generic (PLEG): container finished" podID="95416322-89b0-4f41-b294-508f397deac2" containerID="49f49af8127358b3fe13fef1c162f7da56a1d05236975a577782740cf23b608e" exitCode=0 Nov 03 19:17:03.892150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:03.891743 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" event={"ID":"95416322-89b0-4f41-b294-508f397deac2","Type":"ContainerDied","Data":"49f49af8127358b3fe13fef1c162f7da56a1d05236975a577782740cf23b608e"} Nov 03 19:17:05.062404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.062378 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:05.218023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.217987 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95416322-89b0-4f41-b294-508f397deac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95416322-89b0-4f41-b294-508f397deac2-maas-api-xks-ca\") pod \"95416322-89b0-4f41-b294-508f397deac2\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " Nov 03 19:17:05.218207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.218135 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95416322-89b0-4f41-b294-508f397deac2-kube-api-access-psxcl\" (UniqueName: \"kubernetes.io/projected/95416322-89b0-4f41-b294-508f397deac2-kube-api-access-psxcl\") pod \"95416322-89b0-4f41-b294-508f397deac2\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " Nov 03 19:17:05.218322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.218300 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95416322-89b0-4f41-b294-508f397deac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95416322-89b0-4f41-b294-508f397deac2-maas-api-openshift-ca\") pod \"95416322-89b0-4f41-b294-508f397deac2\" (UID: \"95416322-89b0-4f41-b294-508f397deac2\") " Nov 03 19:17:05.218662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.218638 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95416322-89b0-4f41-b294-508f397deac2-maas-api-openshift-ca" pod "95416322-89b0-4f41-b294-508f397deac2" (UID: "95416322-89b0-4f41-b294-508f397deac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:05.218839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.218817 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95416322-89b0-4f41-b294-508f397deac2-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:05.220242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.220219 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95416322-89b0-4f41-b294-508f397deac2-maas-api-xks-ca" pod "95416322-89b0-4f41-b294-508f397deac2" (UID: "95416322-89b0-4f41-b294-508f397deac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:17:05.220322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.220226 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95416322-89b0-4f41-b294-508f397deac2-kube-api-access-psxcl" pod "95416322-89b0-4f41-b294-508f397deac2" (UID: "95416322-89b0-4f41-b294-508f397deac2"). InnerVolumeSpecName "kube-api-access-psxcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:05.320231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.320193 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95416322-89b0-4f41-b294-508f397deac2-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:05.320231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.320225 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxcl\" (UniqueName: \"kubernetes.io/projected/95416322-89b0-4f41-b294-508f397deac2-kube-api-access-psxcl\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:05.911133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.911097 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" event={"ID":"95416322-89b0-4f41-b294-508f397deac2","Type":"ContainerDied","Data":"61061f26b3695ecc5f7cbb9cb0817605122831abe3c38a49f8165a84498e9992"} Nov 03 19:17:05.911352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.911158 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61061f26b3695ecc5f7cbb9cb0817605122831abe3c38a49f8165a84498e9992" Nov 03 19:17:05.911352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:05.911113 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts" Nov 03 19:17:06.386815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.386790 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:17:06.536024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.535936 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-certs\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.535980 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-kube-api-access-pq8xv\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-kube-api-access-pq8xv\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536053 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-podinfo\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536084 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-socket\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536201 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-token\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-token\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536279 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-envoy\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536312 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-certs" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:06.536437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536388 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istiod-ca-cert\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536439 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-credential-socket\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536530 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-data\") pod \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\" (UID: \"44aa37b6-f6ca-4281-a298-e96c0f0417aa\") " Nov 03 19:17:06.536762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536438 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-socket" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:06.536832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.536766 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istiod-ca-cert" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:06.537147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.537050 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-data" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:06.537279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.537255 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-credential-socket" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:06.537866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.537839 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.537866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.537865 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.538037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.537879 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.538037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.537891 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.538037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.537906 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.545279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.538746 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-token" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:06.545279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.538756 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-kube-api-access-pq8xv" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "kube-api-access-pq8xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:06.545279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.538842 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-podinfo" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:17:06.545279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.538928 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-envoy" pod "44aa37b6-f6ca-4281-a298-e96c0f0417aa" (UID: "44aa37b6-f6ca-4281-a298-e96c0f0417aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:17:06.638999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.638970 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.639186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.639007 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.639186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.639022 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq8xv\" (UniqueName: \"kubernetes.io/projected/44aa37b6-f6ca-4281-a298-e96c0f0417aa-kube-api-access-pq8xv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.639186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.639036 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44aa37b6-f6ca-4281-a298-e96c0f0417aa-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:06.922956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.922856 4948 generic.go:356] "Generic (PLEG): container finished" podID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerID="f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49" exitCode=0 Nov 03 19:17:06.922956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.922925 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" Nov 03 19:17:06.923212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.922952 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" event={"ID":"44aa37b6-f6ca-4281-a298-e96c0f0417aa","Type":"ContainerDied","Data":"f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49"} Nov 03 19:17:06.923212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.923014 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g" event={"ID":"44aa37b6-f6ca-4281-a298-e96c0f0417aa","Type":"ContainerDied","Data":"5eb74aede5de079cea4611595b930a45dfc6828643c0bdc479f3a58ea49a7faf"} Nov 03 19:17:06.923212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.923027 4948 scope.go:122] "RemoveContainer" containerID="f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49" Nov 03 19:17:06.934597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.934576 4948 scope.go:122] "RemoveContainer" containerID="5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193" Nov 03 19:17:06.945108 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.945082 4948 scope.go:122] "RemoveContainer" containerID="f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49" Nov 03 19:17:06.945429 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:06.945409 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49\": container with ID starting with f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49 not found: ID does not exist" containerID="f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49" Nov 03 19:17:06.945512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.945439 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49"} err="failed to get container status \"f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49\": rpc error: code = NotFound desc = could not find container \"f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49\": container with ID starting with f88ed8eb1f0315df5638e299e1ff0a541187b5ae8846841821aca895e5b95c49 not found: ID does not exist" Nov 03 19:17:06.945512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.945456 4948 scope.go:122] "RemoveContainer" containerID="5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193" Nov 03 19:17:06.945769 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:06.945743 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193\": container with ID starting with 5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193 not found: ID does not exist" containerID="5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193" Nov 03 19:17:06.945861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:06.945774 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193"} err="failed to get container status \"5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193\": rpc error: code = NotFound desc = could not find container \"5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193\": container with ID starting with 5231f1d482ec3b5950d02e8b1beb1b8046f24d49d1abdc4580c9cd872fc96193 not found: ID does not exist" Nov 03 19:17:07.094538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:07.094501 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g"] Nov 03 19:17:07.145732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:07.145663 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ba5674-openshift-default-767964dd8c-9x84g"] Nov 03 19:17:08.754863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:08.754831 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" path="/var/lib/kubelet/pods/44aa37b6-f6ca-4281-a298-e96c0f0417aa/volumes" Nov 03 19:17:13.378234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:13.378198 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f"] Nov 03 19:17:18.378743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.378655 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" podUID="8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" containerName="payload-processing" containerID="cri-o://3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5" gracePeriod=25 Nov 03 19:17:18.656067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.656032 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:17:18.714435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.714400 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-plugins-config-volume\") pod \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " Nov 03 19:17:18.714614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.714519 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-kube-api-access-ljqmh\" (UniqueName: \"kubernetes.io/projected/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-kube-api-access-ljqmh\") pod \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\" (UID: \"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8\") " Nov 03 19:17:18.714821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.714799 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-plugins-config-volume" pod "8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" (UID: "8cbfb1bf-939d-4ef0-a851-f9e35ed780b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:18.714955 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.714936 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:18.716586 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.716557 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-kube-api-access-ljqmh" pod "8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" (UID: "8cbfb1bf-939d-4ef0-a851-f9e35ed780b8"). InnerVolumeSpecName "kube-api-access-ljqmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:18.816373 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:18.816333 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljqmh\" (UniqueName: \"kubernetes.io/projected/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8-kube-api-access-ljqmh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:19.035493 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.035461 4948 generic.go:356] "Generic (PLEG): container finished" podID="8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" containerID="3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5" exitCode=0 Nov 03 19:17:19.035680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.035513 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" Nov 03 19:17:19.035680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.035547 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" event={"ID":"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8","Type":"ContainerDied","Data":"3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5"} Nov 03 19:17:19.035680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.035605 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f" event={"ID":"8cbfb1bf-939d-4ef0-a851-f9e35ed780b8","Type":"ContainerDied","Data":"572553d29558983762ec22fb57802f032ebee9dc9f25a1d9564b7338e0b67afe"} Nov 03 19:17:19.035680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.035628 4948 scope.go:122] "RemoveContainer" containerID="3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5" Nov 03 19:17:19.046640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.046624 4948 scope.go:122] "RemoveContainer" containerID="3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5" Nov 03 19:17:19.046990 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:19.046970 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5\": container with ID starting with 3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5 not found: ID does not exist" containerID="3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5" Nov 03 19:17:19.047065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.046998 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5"} err="failed to get container status \"3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5\": rpc error: code = NotFound desc = could not find container \"3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5\": container with ID starting with 3d43323f1f2e4cdf2142db0b6a29e3b207530732afce111ad80befc41ceb68d5 not found: ID does not exist" Nov 03 19:17:19.206341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.206303 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f"] Nov 03 19:17:19.257127 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:19.257088 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efd81c-768b7754c9-hrk6f"] Nov 03 19:17:20.756280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:20.756246 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" path="/var/lib/kubelet/pods/8cbfb1bf-939d-4ef0-a851-f9e35ed780b8/volumes" Nov 03 19:17:32.794188 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:32.794106 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:17:38.104210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104171 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm"] Nov 03 19:17:38.104829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104814 4948 cpu_manager.go:414] "containerMap: removing container" podUID="95416322-89b0-4f41-b294-508f397deac2" containerName="revoke-keys" Nov 03 19:17:38.104891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104831 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95416322-89b0-4f41-b294-508f397deac2" containerName="revoke-keys" Nov 03 19:17:38.104891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104866 4948 cpu_manager.go:414] "containerMap: removing container" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" Nov 03 19:17:38.104891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104871 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" Nov 03 19:17:38.104891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104887 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" containerName="payload-processing" Nov 03 19:17:38.104891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104893 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" containerName="payload-processing" Nov 03 19:17:38.105059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104900 4948 cpu_manager.go:414] "containerMap: removing container" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" Nov 03 19:17:38.105059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104905 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" Nov 03 19:17:38.105059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104989 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" Nov 03 19:17:38.105059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.104999 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbfb1bf-939d-4ef0-a851-f9e35ed780b8" containerName="payload-processing" Nov 03 19:17:38.105059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.105008 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="95416322-89b0-4f41-b294-508f397deac2" containerName="revoke-keys" Nov 03 19:17:38.110193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.110171 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.155875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.155837 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f"] Nov 03 19:17:38.156718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.156679 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="44aa37b6-f6ca-4281-a298-e96c0f0417aa" containerName="istio-proxy" Nov 03 19:17:38.160311 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.160288 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p"] Nov 03 19:17:38.160429 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.160411 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.164688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.164667 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.202491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.202457 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h6x4t\"" Nov 03 19:17:38.207220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.207194 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm"] Nov 03 19:17:38.255837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.255801 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f"] Nov 03 19:17:38.260119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260086 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p"] Nov 03 19:17:38.260578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260547 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.260674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260613 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.260763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260692 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.260825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260792 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.260825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260815 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.260941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260841 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.260972 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261011 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.261119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261069 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nfc\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-kube-api-access-z5nfc\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261129 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.261172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261152 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szs84\" (UniqueName: \"kubernetes.io/projected/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-kube-api-access-szs84\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.261282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261182 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.261282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261240 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.261393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261290 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.261393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261307 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261329 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9a816-c24a-49a1-bdda-20144fa2d008-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261374 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261447 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.261611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261466 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261547 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwk8\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-kube-api-access-lzwk8\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.261611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261577 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.261885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261628 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261669 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.261885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261689 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.261885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261749 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.261885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261765 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.261885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.261783 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.363537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363441 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nfc\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-kube-api-access-z5nfc\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.363537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363491 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.363537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363514 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szs84\" (UniqueName: \"kubernetes.io/projected/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-kube-api-access-szs84\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.363886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363560 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.363886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363586 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.363886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363734 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.363886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363787 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.363886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363822 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9a816-c24a-49a1-bdda-20144fa2d008-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.363886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363852 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363899 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363925 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363968 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.363997 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364047 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwk8\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-kube-api-access-lzwk8\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364079 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364110 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364177 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364211 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364259 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364314 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364318 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364342 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364401 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364451 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364484 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364558 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364568 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364589 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364620 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364633 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364715 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364725 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9a816-c24a-49a1-bdda-20144fa2d008-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.364774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.364749 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.365612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.365190 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.365612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.365303 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.365757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.365693 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.366148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.366079 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.366148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.366108 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.366148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.366138 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.366433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.366402 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.366572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.366541 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.366717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.366681 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.367678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.367643 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.367824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.367669 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.368311 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.368286 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.368797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.368775 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.369116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.369091 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.369868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.369842 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.428291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.428257 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szs84\" (UniqueName: \"kubernetes.io/projected/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-kube-api-access-szs84\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.428291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.428283 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.428489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.428341 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwk8\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-kube-api-access-lzwk8\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.428489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.428377 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8n78f\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.428489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.428392 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nfc\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-kube-api-access-z5nfc\") pod \"maas-default-gateway-openshift-default-7854949c9-npq2p\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.433672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.433649 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a8205c-e0cd-4eeb-828a-c78ec53fe35b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-vdlrm\" (UID: \"99a8205c-e0cd-4eeb-828a-c78ec53fe35b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.477391 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.477355 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:38.487434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.487239 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:38.724862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.724826 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:38.872666 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:17:38.872638 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b28c18_efa0_4d8f_8b3f_92a2ed348058.slice/crio-4fb6e030067625dc1cb12cdc5e11b9e3a1bc8decf4bdf72d4dea39386d53039b WatchSource:0}: Error finding container 4fb6e030067625dc1cb12cdc5e11b9e3a1bc8decf4bdf72d4dea39386d53039b: Status 404 returned error can't find the container with id 4fb6e030067625dc1cb12cdc5e11b9e3a1bc8decf4bdf72d4dea39386d53039b Nov 03 19:17:38.872925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.872901 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f"] Nov 03 19:17:38.874936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.874904 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:38.875050 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.874975 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:38.875050 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.875023 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:38.922677 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:17:38.922646 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa9a816_c24a_49a1_bdda_20144fa2d008.slice/crio-c5cc1af9fb6adbed1a46a6388e7a3eff001fac582a2a1c02b8671e9f91264232 WatchSource:0}: Error finding container c5cc1af9fb6adbed1a46a6388e7a3eff001fac582a2a1c02b8671e9f91264232: Status 404 returned error can't find the container with id c5cc1af9fb6adbed1a46a6388e7a3eff001fac582a2a1c02b8671e9f91264232 Nov 03 19:17:38.925300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.925273 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p"] Nov 03 19:17:38.925605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.925573 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:38.925714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.925650 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:38.925714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:38.925684 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:39.118543 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:17:39.118510 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a8205c_e0cd_4eeb_828a_c78ec53fe35b.slice/crio-4fdf0244f5c4e8d3e5bed708926129edf45376544e54e428360ad754374fe861 WatchSource:0}: Error finding container 4fdf0244f5c4e8d3e5bed708926129edf45376544e54e428360ad754374fe861: Status 404 returned error can't find the container with id 4fdf0244f5c4e8d3e5bed708926129edf45376544e54e428360ad754374fe861 Nov 03 19:17:39.119568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.119536 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm"] Nov 03 19:17:39.121962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.121930 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:39.122047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.122003 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:39.122085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.122043 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:17:39.204986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.204945 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" event={"ID":"8aa9a816-c24a-49a1-bdda-20144fa2d008","Type":"ContainerStarted","Data":"27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb"} Nov 03 19:17:39.205174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.205022 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" event={"ID":"8aa9a816-c24a-49a1-bdda-20144fa2d008","Type":"ContainerStarted","Data":"c5cc1af9fb6adbed1a46a6388e7a3eff001fac582a2a1c02b8671e9f91264232"} Nov 03 19:17:39.207308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.207280 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" event={"ID":"49b28c18-efa0-4d8f-8b3f-92a2ed348058","Type":"ContainerStarted","Data":"f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76"} Nov 03 19:17:39.207422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.207339 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" event={"ID":"49b28c18-efa0-4d8f-8b3f-92a2ed348058","Type":"ContainerStarted","Data":"4fb6e030067625dc1cb12cdc5e11b9e3a1bc8decf4bdf72d4dea39386d53039b"} Nov 03 19:17:39.209620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.209596 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" event={"ID":"99a8205c-e0cd-4eeb-828a-c78ec53fe35b","Type":"ContainerStarted","Data":"4fdf0244f5c4e8d3e5bed708926129edf45376544e54e428360ad754374fe861"} Nov 03 19:17:39.342327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.342212 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podStartSLOduration=1.342189068 podStartE2EDuration="1.342189068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:39.328234577 +0000 UTC m=+2388.811230504" watchObservedRunningTime="2026-11-03 19:17:39.342189068 +0000 UTC m=+2388.825184983" Nov 03 19:17:39.462722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.461754 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podStartSLOduration=1.461730682 podStartE2EDuration="1.461730682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:39.450824538 +0000 UTC m=+2388.933820453" watchObservedRunningTime="2026-11-03 19:17:39.461730682 +0000 UTC m=+2388.944726595" Nov 03 19:17:39.481674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.480397 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:39.484222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.482000 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:39.484222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.482071 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:39.487371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:39.487338 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:40.245867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.245805 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" event={"ID":"99a8205c-e0cd-4eeb-828a-c78ec53fe35b","Type":"ContainerStarted","Data":"a037e39f2443587d88e83bb6e281dbed05c3a98c226a33e9683e4cd536ff1719"} Nov 03 19:17:40.390728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.384427 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podStartSLOduration=2.3844023979999998 podStartE2EDuration="2.384402398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:40.369234338 +0000 UTC m=+2389.852230252" watchObservedRunningTime="2026-11-03 19:17:40.384402398 +0000 UTC m=+2389.867398313" Nov 03 19:17:40.484246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.484209 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:40.484527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.484496 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:40.502037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.501931 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:40.502314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.502280 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:40.732724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.729641 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:40.732724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.732150 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:40.732724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:40.732218 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:41.482141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:41.482105 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:41.482813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:41.482782 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:41.489113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:41.489082 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:41.489345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:41.489320 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:41.726538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:41.726100 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:41.726538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:41.726161 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:42.478268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:42.478167 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:42.478574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:42.478519 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:42.494168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:42.494129 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:42.494788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:42.494755 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:42.725727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:42.725257 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:42.725727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:42.725320 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:43.478723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:43.478535 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:43.478723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:43.478609 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:43.494724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:43.491185 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:43.494724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:43.491252 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:43.732769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:43.727154 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:43.732769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:43.727217 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:44.480731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:44.480671 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:44.481070 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:44.481023 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:44.489123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:44.489090 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:44.489372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:44.489342 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:44.747834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:44.741233 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:44.747834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:44.741293 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:45.484724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:45.482114 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:45.484724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:45.482178 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:45.494723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:45.490060 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:45.494723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:45.490120 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:45.728740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:45.728562 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:45.728740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:45.728642 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:46.478373 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:46.478331 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:46.479030 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:46.478974 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:46.487853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:46.487817 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:46.488041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:46.487879 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:46.727016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:46.726188 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:46.727016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:46.726266 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:47.477830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:47.477786 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8n78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 19:17:47.478062 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:47.477863 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 19:17:47.488333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:47.488291 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-npq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 19:17:47.488897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:47.488362 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 19:17:47.725769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:47.725241 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 19:17:47.725769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:47.725303 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 19:17:48.911613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:48.911574 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:48.912787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:48.912766 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:48.913761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:48.913741 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:17:48.928890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:48.928839 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:48.930451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:48.930429 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:48.933096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:48.933072 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:17:49.055155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:49.055101 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:49.055585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:49.055566 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:49.056578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:49.056561 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" Nov 03 19:17:51.943279 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:51.943239 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk"] Nov 03 19:17:51.943813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:51.943693 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" podUID="bcefc690-86a7-4d07-888d-682eb583ce92" containerName="maas-api" containerID="cri-o://0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c" gracePeriod=30 Nov 03 19:17:52.356252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.355983 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd"] Nov 03 19:17:52.364957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.364707 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.365349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.365324 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:17:52.406797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.406754 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd"] Nov 03 19:17:52.418771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.418733 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-942af8\"" Nov 03 19:17:52.418771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.418764 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-942af8-dockercfg-tltjc\"" Nov 03 19:17:52.447335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.447295 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcefc690-86a7-4d07-888d-682eb583ce92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc690-86a7-4d07-888d-682eb583ce92-openshift-service-ca\") pod \"bcefc690-86a7-4d07-888d-682eb583ce92\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " Nov 03 19:17:52.447543 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.447436 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls\") pod \"bcefc690-86a7-4d07-888d-682eb583ce92\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " Nov 03 19:17:52.447632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.447616 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcefc690-86a7-4d07-888d-682eb583ce92-kube-api-access-4jcxw\" (UniqueName: \"kubernetes.io/projected/bcefc690-86a7-4d07-888d-682eb583ce92-kube-api-access-4jcxw\") pod \"bcefc690-86a7-4d07-888d-682eb583ce92\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " Nov 03 19:17:52.447728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.447673 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-metrics-certs\") pod \"bcefc690-86a7-4d07-888d-682eb583ce92\" (UID: \"bcefc690-86a7-4d07-888d-682eb583ce92\") " Nov 03 19:17:52.447802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.447743 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcefc690-86a7-4d07-888d-682eb583ce92-openshift-service-ca" pod "bcefc690-86a7-4d07-888d-682eb583ce92" (UID: "bcefc690-86a7-4d07-888d-682eb583ce92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:52.447948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.447925 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvz8k\" (UniqueName: \"kubernetes.io/projected/b8d74572-40d3-4a32-a5af-91349b852192-kube-api-access-jvz8k\") pod \"payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.448103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.448086 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d74572-40d3-4a32-a5af-91349b852192-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.448395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.448374 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc690-86a7-4d07-888d-682eb583ce92-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:52.449814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.449788 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcefc690-86a7-4d07-888d-682eb583ce92-kube-api-access-4jcxw" pod "bcefc690-86a7-4d07-888d-682eb583ce92" (UID: "bcefc690-86a7-4d07-888d-682eb583ce92"). InnerVolumeSpecName "kube-api-access-4jcxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:52.450090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.450068 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls" pod "bcefc690-86a7-4d07-888d-682eb583ce92" (UID: "bcefc690-86a7-4d07-888d-682eb583ce92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:17:52.450090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.450082 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-metrics-certs" pod "bcefc690-86a7-4d07-888d-682eb583ce92" (UID: "bcefc690-86a7-4d07-888d-682eb583ce92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:17:52.562814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.558630 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d74572-40d3-4a32-a5af-91349b852192-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.562814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.558895 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvz8k\" (UniqueName: \"kubernetes.io/projected/b8d74572-40d3-4a32-a5af-91349b852192-kube-api-access-jvz8k\") pod \"payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.562814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.558976 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jcxw\" (UniqueName: \"kubernetes.io/projected/bcefc690-86a7-4d07-888d-682eb583ce92-kube-api-access-4jcxw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:52.562814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.558993 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:52.562814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.559007 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcefc690-86a7-4d07-888d-682eb583ce92-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:52.562814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.559988 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d74572-40d3-4a32-a5af-91349b852192-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.658167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.657997 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvz8k\" (UniqueName: \"kubernetes.io/projected/b8d74572-40d3-4a32-a5af-91349b852192-kube-api-access-jvz8k\") pod \"payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.684104 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.683798 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:52.706503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.706454 4948 generic.go:356] "Generic (PLEG): container finished" podID="bcefc690-86a7-4d07-888d-682eb583ce92" containerID="0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c" exitCode=0 Nov 03 19:17:52.706822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.706795 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" event={"ID":"bcefc690-86a7-4d07-888d-682eb583ce92","Type":"ContainerDied","Data":"0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c"} Nov 03 19:17:52.707074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.707053 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" event={"ID":"bcefc690-86a7-4d07-888d-682eb583ce92","Type":"ContainerDied","Data":"52a95a0670f9c496363744d8577e059221ed6d48ab361bd490d6532913b796b9"} Nov 03 19:17:52.707234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.707218 4948 scope.go:122] "RemoveContainer" containerID="0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c" Nov 03 19:17:52.707576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.707547 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk" Nov 03 19:17:52.731218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.731188 4948 scope.go:122] "RemoveContainer" containerID="0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c" Nov 03 19:17:52.731982 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:52.731909 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c\": container with ID starting with 0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c not found: ID does not exist" containerID="0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c" Nov 03 19:17:52.731982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.731952 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c"} err="failed to get container status \"0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c\": rpc error: code = NotFound desc = could not find container \"0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c\": container with ID starting with 0b863b305363012128cc132e27bf20437c4a98d6d611a9ad8e307cc0c1053e3c not found: ID does not exist" Nov 03 19:17:52.893739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.893611 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk"] Nov 03 19:17:52.899961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:52.899930 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e1f8f8-f49765459-65qrk"] Nov 03 19:17:53.129826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:53.129655 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd"] Nov 03 19:17:53.286053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:53.286018 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn"] Nov 03 19:17:53.293245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:53.293216 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65e151-6lpsn"] Nov 03 19:17:53.717976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:53.717941 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" event={"ID":"b8d74572-40d3-4a32-a5af-91349b852192","Type":"ContainerStarted","Data":"d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334"} Nov 03 19:17:53.718172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:53.718006 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" event={"ID":"b8d74572-40d3-4a32-a5af-91349b852192","Type":"ContainerStarted","Data":"cb7c02687d62527d4a8e2d3f893b0bf3f5f8bea5e8295a93aee59f137d515ed3"} Nov 03 19:17:53.718172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:53.718034 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:17:53.848798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:53.848729 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" podStartSLOduration=1.8486949529999999 podStartE2EDuration="1.848694953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:53.846239974 +0000 UTC m=+2403.329235888" watchObservedRunningTime="2026-11-03 19:17:53.848694953 +0000 UTC m=+2403.331690869" Nov 03 19:17:54.423215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:54.423177 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk"] Nov 03 19:17:54.474248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:54.474214 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1f8f8-tr2hk"] Nov 03 19:17:54.768296 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:54.768251 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54703451-d2e7-45b6-9040-a6f86940e78a" path="/var/lib/kubelet/pods/54703451-d2e7-45b6-9040-a6f86940e78a/volumes" Nov 03 19:17:54.768943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:54.768923 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b909622-2a7b-48de-b2cc-50bc64b60f57" path="/var/lib/kubelet/pods/7b909622-2a7b-48de-b2cc-50bc64b60f57/volumes" Nov 03 19:17:54.769509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:54.769494 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcefc690-86a7-4d07-888d-682eb583ce92" path="/var/lib/kubelet/pods/bcefc690-86a7-4d07-888d-682eb583ce92/volumes" Nov 03 19:17:55.579166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:55.579134 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:17:55.586907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:55.586885 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:17:55.846857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:55.846772 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:17:55.854288 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:55.854257 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:17:58.763643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.763606 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96"] Nov 03 19:17:58.764254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.764240 4948 cpu_manager.go:414] "containerMap: removing container" podUID="bcefc690-86a7-4d07-888d-682eb583ce92" containerName="maas-api" Nov 03 19:17:58.764296 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.764255 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcefc690-86a7-4d07-888d-682eb583ce92" containerName="maas-api" Nov 03 19:17:58.764402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.764392 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcefc690-86a7-4d07-888d-682eb583ce92" containerName="maas-api" Nov 03 19:17:58.769434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.769417 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.853333 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.853286 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-openshift-service-ca\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.853519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.853496 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.853612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.853566 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmgk\" (UniqueName: \"kubernetes.io/projected/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-kube-api-access-9cmgk\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.853752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.853711 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-metrics-certs\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.883256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.883228 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d74f801-serving-cert\"" Nov 03 19:17:58.887801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.887771 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96"] Nov 03 19:17:58.955429 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.955386 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-metrics-certs\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.955619 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.955602 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-openshift-service-ca\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.955737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.955719 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.955825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.955773 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmgk\" (UniqueName: \"kubernetes.io/projected/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-kube-api-access-9cmgk\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.955918 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:58.955895 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-serving-cert: secret "maas-api-e2e-mt-3d74f801-serving-cert" not found Nov 03 19:17:58.955977 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:17:58.955967 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls podName:e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f nodeName:}" failed. No retries permitted until 2026-11-03 19:17:59.455948362 +0000 UTC m=+2408.938944259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls") pod "maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" (UID: "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f") : secret "maas-api-e2e-mt-3d74f801-serving-cert" not found Nov 03 19:17:58.956403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.956382 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-openshift-service-ca\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:58.958228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:58.958209 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-metrics-certs\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:59.020571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:59.020495 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmgk\" (UniqueName: \"kubernetes.io/projected/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-kube-api-access-9cmgk\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:59.494725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:59.482578 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:59.557720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:59.555467 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:17:59.684472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:17:59.684436 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:18:00.095732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:00.094998 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96"] Nov 03 19:18:00.873297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:00.873244 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" event={"ID":"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f","Type":"ContainerStarted","Data":"6054cbdbf5b240a56472d2e629e75d3a1ec9275ee669dede2842f247667cf225"} Nov 03 19:18:01.543207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:01.543168 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58"] Nov 03 19:18:01.543666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:01.543407 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" containerID="cri-o://8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02" gracePeriod=30 Nov 03 19:18:01.885023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:01.884933 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" event={"ID":"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f","Type":"ContainerStarted","Data":"211e43d8387d0ab870142f0bbd236952c47b58e0695512f137f9603cba99202b"} Nov 03 19:18:01.885177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:01.885030 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:18:02.009370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.009312 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" podStartSLOduration=3.097140454 podStartE2EDuration="4.009296512s" totalImagesPullingTime="912.156058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:02.005654936 +0000 UTC m=+2411.488650849" watchObservedRunningTime="2026-11-03 19:18:02.009296512 +0000 UTC m=+2411.492292425" Nov 03 19:18:02.378507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.378468 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw"] Nov 03 19:18:02.382896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.382877 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.384620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.384590 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.384802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.384781 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jm6r\" (UniqueName: \"kubernetes.io/projected/9c51611c-8f6f-4fc3-988c-93b1a25427a2-kube-api-access-9jm6r\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.384862 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.384853 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.436283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.436240 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:18:02.436520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.436465 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:18:02.436648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.436622 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw"] Nov 03 19:18:02.485662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.485617 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.485848 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.485827 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jm6r\" (UniqueName: \"kubernetes.io/projected/9c51611c-8f6f-4fc3-988c-93b1a25427a2-kube-api-access-9jm6r\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.485940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.485923 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.486814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.486780 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.488292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.488268 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.548896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.548862 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jm6r\" (UniqueName: \"kubernetes.io/projected/9c51611c-8f6f-4fc3-988c-93b1a25427a2-kube-api-access-9jm6r\") pod \"maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.701207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:02.701173 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:02.838193 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:02.837927 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:18:03.099741 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:03.099691 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw"] Nov 03 19:18:03.099891 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:18:03.099745 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c51611c_8f6f_4fc3_988c_93b1a25427a2.slice/crio-a8f787602af957f69998c11d1fd17c5fdb0945757c31476b393d245a6ff2b335 WatchSource:0}: Error finding container a8f787602af957f69998c11d1fd17c5fdb0945757c31476b393d245a6ff2b335: Status 404 returned error can't find the container with id a8f787602af957f69998c11d1fd17c5fdb0945757c31476b393d245a6ff2b335 Nov 03 19:18:03.908180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:03.908081 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" event={"ID":"9c51611c-8f6f-4fc3-988c-93b1a25427a2","Type":"ContainerStarted","Data":"880f8dc4648e16d4a972a6f2744d19bcdf6f516bf7a86c9f6505a3b1874b1fd4"} Nov 03 19:18:03.908180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:03.908171 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" event={"ID":"9c51611c-8f6f-4fc3-988c-93b1a25427a2","Type":"ContainerStarted","Data":"a8f787602af957f69998c11d1fd17c5fdb0945757c31476b393d245a6ff2b335"} Nov 03 19:18:04.028028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:04.027972 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" podStartSLOduration=2.027958598 podStartE2EDuration="2.027958598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:04.026111805 +0000 UTC m=+2413.509107720" watchObservedRunningTime="2026-11-03 19:18:04.027958598 +0000 UTC m=+2413.510954512" Nov 03 19:18:04.730814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:04.730780 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:18:04.808454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:04.808415 4948 patch_prober.go:29] interesting pod/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:18:04.808647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:04.808507 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:18:04.918786 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:04.918751 4948 generic.go:356] "Generic (PLEG): container finished" podID="9c51611c-8f6f-4fc3-988c-93b1a25427a2" containerID="880f8dc4648e16d4a972a6f2744d19bcdf6f516bf7a86c9f6505a3b1874b1fd4" exitCode=0 Nov 03 19:18:04.919287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:04.918807 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" event={"ID":"9c51611c-8f6f-4fc3-988c-93b1a25427a2","Type":"ContainerDied","Data":"880f8dc4648e16d4a972a6f2744d19bcdf6f516bf7a86c9f6505a3b1874b1fd4"} Nov 03 19:18:06.125513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.125480 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:06.242161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.242124 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-xks-ca\") pod \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " Nov 03 19:18:06.242350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.242170 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-openshift-ca\") pod \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " Nov 03 19:18:06.242350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.242222 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c51611c-8f6f-4fc3-988c-93b1a25427a2-kube-api-access-9jm6r\" (UniqueName: \"kubernetes.io/projected/9c51611c-8f6f-4fc3-988c-93b1a25427a2-kube-api-access-9jm6r\") pod \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\" (UID: \"9c51611c-8f6f-4fc3-988c-93b1a25427a2\") " Nov 03 19:18:06.242636 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.242609 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-openshift-ca" pod "9c51611c-8f6f-4fc3-988c-93b1a25427a2" (UID: "9c51611c-8f6f-4fc3-988c-93b1a25427a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:06.242955 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.242928 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.244389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.244360 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-xks-ca" pod "9c51611c-8f6f-4fc3-988c-93b1a25427a2" (UID: "9c51611c-8f6f-4fc3-988c-93b1a25427a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:06.244484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.244434 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c51611c-8f6f-4fc3-988c-93b1a25427a2-kube-api-access-9jm6r" pod "9c51611c-8f6f-4fc3-988c-93b1a25427a2" (UID: "9c51611c-8f6f-4fc3-988c-93b1a25427a2"). InnerVolumeSpecName "kube-api-access-9jm6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:06.344876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.344842 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51611c-8f6f-4fc3-988c-93b1a25427a2-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.344876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.344873 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jm6r\" (UniqueName: \"kubernetes.io/projected/9c51611c-8f6f-4fc3-988c-93b1a25427a2-kube-api-access-9jm6r\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.840155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.840127 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:18:06.938171 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.938138 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" Nov 03 19:18:06.938343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.938143 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw" event={"ID":"9c51611c-8f6f-4fc3-988c-93b1a25427a2","Type":"ContainerDied","Data":"a8f787602af957f69998c11d1fd17c5fdb0945757c31476b393d245a6ff2b335"} Nov 03 19:18:06.938343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.938274 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f787602af957f69998c11d1fd17c5fdb0945757c31476b393d245a6ff2b335" Nov 03 19:18:06.940110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.940079 4948 generic.go:356] "Generic (PLEG): container finished" podID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerID="8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02" exitCode=0 Nov 03 19:18:06.940248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.940123 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" Nov 03 19:18:06.940248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.940160 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" event={"ID":"3d57cd9d-8092-445f-8a1a-b8773a31479f","Type":"ContainerDied","Data":"8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02"} Nov 03 19:18:06.940248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.940210 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58" event={"ID":"3d57cd9d-8092-445f-8a1a-b8773a31479f","Type":"ContainerDied","Data":"a70ef35c6614da83fa93a895e37969609f30b786e338b73e07d423e085a34a3b"} Nov 03 19:18:06.940248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.940239 4948 scope.go:122] "RemoveContainer" containerID="8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02" Nov 03 19:18:06.952851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.952831 4948 scope.go:122] "RemoveContainer" containerID="be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1" Nov 03 19:18:06.953275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953255 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-kube-api-access-892lm\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-kube-api-access-892lm\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953306 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d57cd9d-8092-445f-8a1a-b8773a31479f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d57cd9d-8092-445f-8a1a-b8773a31479f-istiod-ca-cert\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953437 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-socket\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953473 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-envoy\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953562 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953509 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-token\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-token\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953557 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-certs\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953596 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-credential-socket\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953750 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-podinfo\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953807 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-data\") pod \"3d57cd9d-8092-445f-8a1a-b8773a31479f\" (UID: \"3d57cd9d-8092-445f-8a1a-b8773a31479f\") " Nov 03 19:18:06.953882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953827 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-socket" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:06.953882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.953848 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d57cd9d-8092-445f-8a1a-b8773a31479f-istiod-ca-cert" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:06.954194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.954170 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-credential-socket" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:06.954868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.954635 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-certs" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:06.954868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.954712 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d57cd9d-8092-445f-8a1a-b8773a31479f-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.954868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.954736 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.954868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.954754 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.954868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.954823 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-data" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:06.955884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.955863 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-kube-api-access-892lm" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "kube-api-access-892lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:06.956427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.956389 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-envoy" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:06.956799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.956773 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-podinfo" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:18:06.956928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.956910 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-token" pod "3d57cd9d-8092-445f-8a1a-b8773a31479f" (UID: "3d57cd9d-8092-445f-8a1a-b8773a31479f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:06.964971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.964950 4948 scope.go:122] "RemoveContainer" containerID="8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02" Nov 03 19:18:06.965334 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:06.965300 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02\": container with ID starting with 8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02 not found: ID does not exist" containerID="8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02" Nov 03 19:18:06.965413 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.965341 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02"} err="failed to get container status \"8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02\": rpc error: code = NotFound desc = could not find container \"8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02\": container with ID starting with 8aa75e58646460dac86025790abae20be17dc50b82e7351a4e771d83a8a73b02 not found: ID does not exist" Nov 03 19:18:06.965413 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.965365 4948 scope.go:122] "RemoveContainer" containerID="be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1" Nov 03 19:18:06.965671 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:06.965647 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1\": container with ID starting with be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1 not found: ID does not exist" containerID="be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1" Nov 03 19:18:06.965751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:06.965681 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1"} err="failed to get container status \"be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1\": rpc error: code = NotFound desc = could not find container \"be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1\": container with ID starting with be9427a267adfba5704f89821e75ab323530cf6c468e2fac4b1701ef88a9c2b1 not found: ID does not exist" Nov 03 19:18:07.056468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.056428 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:07.056468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.056471 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:07.056743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.056486 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-892lm\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-kube-api-access-892lm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:07.056743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.056502 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:07.056743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.056515 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d57cd9d-8092-445f-8a1a-b8773a31479f-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:07.056743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.056528 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d57cd9d-8092-445f-8a1a-b8773a31479f-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:07.373303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.373205 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58"] Nov 03 19:18:07.379769 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.379729 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e1f8f8-openshift-default-676bdb65df-bsq58"] Nov 03 19:18:07.593344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.593302 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc"] Nov 03 19:18:07.902431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:07.902403 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:18:08.757291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:08.757240 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" path="/var/lib/kubelet/pods/3d57cd9d-8092-445f-8a1a-b8773a31479f/volumes" Nov 03 19:18:10.492955 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.492915 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf"] Nov 03 19:18:10.493871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.493848 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" Nov 03 19:18:10.494009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.493871 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" Nov 03 19:18:10.494009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.493907 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9c51611c-8f6f-4fc3-988c-93b1a25427a2" containerName="revoke-keys" Nov 03 19:18:10.494009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.493915 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c51611c-8f6f-4fc3-988c-93b1a25427a2" containerName="revoke-keys" Nov 03 19:18:10.494009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.493936 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" Nov 03 19:18:10.494009 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.493944 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" Nov 03 19:18:10.494283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.494117 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c51611c-8f6f-4fc3-988c-93b1a25427a2" containerName="revoke-keys" Nov 03 19:18:10.494283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.494131 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" Nov 03 19:18:10.499462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.499429 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.551775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.551736 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf"] Nov 03 19:18:10.562991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.562937 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:18:10.596984 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.596950 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:18:10.608182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.608148 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.608376 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.608283 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzv7\" (UniqueName: \"kubernetes.io/projected/f84c93f8-40ee-4fdc-8530-2ed0885f70da-kube-api-access-qtzv7\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.608452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.608382 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.710151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.710118 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.710338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.710223 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzv7\" (UniqueName: \"kubernetes.io/projected/f84c93f8-40ee-4fdc-8530-2ed0885f70da-kube-api-access-qtzv7\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.710338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.710322 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.710844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.710820 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.712736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.712715 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.772485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.772409 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzv7\" (UniqueName: \"kubernetes.io/projected/f84c93f8-40ee-4fdc-8530-2ed0885f70da-kube-api-access-qtzv7\") pod \"maas-api-revoke-sub-7da52e3b-xt2rf\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:10.814948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:10.814917 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:11.200214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:11.200180 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf"] Nov 03 19:18:11.201491 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:18:11.201463 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84c93f8_40ee_4fdc_8530_2ed0885f70da.slice/crio-b0ab7ca63e5793b162004a340785bca16d385a1f583ac56bacd750652cbde781 WatchSource:0}: Error finding container b0ab7ca63e5793b162004a340785bca16d385a1f583ac56bacd750652cbde781: Status 404 returned error can't find the container with id b0ab7ca63e5793b162004a340785bca16d385a1f583ac56bacd750652cbde781 Nov 03 19:18:11.992385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:11.992346 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" event={"ID":"f84c93f8-40ee-4fdc-8530-2ed0885f70da","Type":"ContainerStarted","Data":"8ab3e52328cd4a1643d1dd58d1e1c92fc7367a6dc1dbc2cc4464746ad54d2321"} Nov 03 19:18:11.992896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:11.992429 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" event={"ID":"f84c93f8-40ee-4fdc-8530-2ed0885f70da","Type":"ContainerStarted","Data":"b0ab7ca63e5793b162004a340785bca16d385a1f583ac56bacd750652cbde781"} Nov 03 19:18:12.111791 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:12.111734 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" podStartSLOduration=2.111717281 podStartE2EDuration="2.111717281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:12.109679872 +0000 UTC m=+2421.592675784" watchObservedRunningTime="2026-11-03 19:18:12.111717281 +0000 UTC m=+2421.594713186" Nov 03 19:18:13.001929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:13.001895 4948 generic.go:356] "Generic (PLEG): container finished" podID="f84c93f8-40ee-4fdc-8530-2ed0885f70da" containerID="8ab3e52328cd4a1643d1dd58d1e1c92fc7367a6dc1dbc2cc4464746ad54d2321" exitCode=0 Nov 03 19:18:13.002350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:13.001952 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" event={"ID":"f84c93f8-40ee-4fdc-8530-2ed0885f70da","Type":"ContainerDied","Data":"8ab3e52328cd4a1643d1dd58d1e1c92fc7367a6dc1dbc2cc4464746ad54d2321"} Nov 03 19:18:13.912947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:13.912906 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp"] Nov 03 19:18:14.188250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.188217 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:14.255339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.255302 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-openshift-ca\") pod \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " Nov 03 19:18:14.255502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.255373 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84c93f8-40ee-4fdc-8530-2ed0885f70da-kube-api-access-qtzv7\" (UniqueName: \"kubernetes.io/projected/f84c93f8-40ee-4fdc-8530-2ed0885f70da-kube-api-access-qtzv7\") pod \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " Nov 03 19:18:14.255502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.255452 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-xks-ca\") pod \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\" (UID: \"f84c93f8-40ee-4fdc-8530-2ed0885f70da\") " Nov 03 19:18:14.255803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.255773 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-openshift-ca" pod "f84c93f8-40ee-4fdc-8530-2ed0885f70da" (UID: "f84c93f8-40ee-4fdc-8530-2ed0885f70da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:14.255949 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.255935 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:14.257629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.257604 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-xks-ca" pod "f84c93f8-40ee-4fdc-8530-2ed0885f70da" (UID: "f84c93f8-40ee-4fdc-8530-2ed0885f70da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:14.257755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.257610 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84c93f8-40ee-4fdc-8530-2ed0885f70da-kube-api-access-qtzv7" pod "f84c93f8-40ee-4fdc-8530-2ed0885f70da" (UID: "f84c93f8-40ee-4fdc-8530-2ed0885f70da"). InnerVolumeSpecName "kube-api-access-qtzv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:14.370732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.361311 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtzv7\" (UniqueName: \"kubernetes.io/projected/f84c93f8-40ee-4fdc-8530-2ed0885f70da-kube-api-access-qtzv7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:14.370732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:14.361350 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c93f8-40ee-4fdc-8530-2ed0885f70da-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:15.030606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:15.030519 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" Nov 03 19:18:15.030606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:15.030539 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf" event={"ID":"f84c93f8-40ee-4fdc-8530-2ed0885f70da","Type":"ContainerDied","Data":"b0ab7ca63e5793b162004a340785bca16d385a1f583ac56bacd750652cbde781"} Nov 03 19:18:15.030838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:15.030635 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0ab7ca63e5793b162004a340785bca16d385a1f583ac56bacd750652cbde781" Nov 03 19:18:18.913311 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:18.913256 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" podUID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" containerName="payload-processing" containerID="cri-o://7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d" gracePeriod=25 Nov 03 19:18:19.704738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:19.704683 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts"] Nov 03 19:18:19.755505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:19.755466 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efd81c-9s5ts"] Nov 03 19:18:20.767582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:20.767543 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95416322-89b0-4f41-b294-508f397deac2" path="/var/lib/kubelet/pods/95416322-89b0-4f41-b294-508f397deac2/volumes" Nov 03 19:18:21.808255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.808221 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w"] Nov 03 19:18:21.809141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.809118 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f84c93f8-40ee-4fdc-8530-2ed0885f70da" containerName="revoke-keys" Nov 03 19:18:21.809141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.809141 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84c93f8-40ee-4fdc-8530-2ed0885f70da" containerName="revoke-keys" Nov 03 19:18:21.809343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.809332 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d57cd9d-8092-445f-8a1a-b8773a31479f" containerName="istio-proxy" Nov 03 19:18:21.809401 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.809351 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84c93f8-40ee-4fdc-8530-2ed0885f70da" containerName="revoke-keys" Nov 03 19:18:21.813100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.813079 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.868491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.868457 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w"] Nov 03 19:18:21.885155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.885118 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.885352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.885168 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.885503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.885472 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zq4b\" (UniqueName: \"kubernetes.io/projected/a76af0b8-8b2e-4be5-821b-c4febd391a80-kube-api-access-6zq4b\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.915270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.915235 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:18:21.915480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.915302 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:18:21.986841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.986802 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.986841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.986851 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.987132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.986939 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zq4b\" (UniqueName: \"kubernetes.io/projected/a76af0b8-8b2e-4be5-821b-c4febd391a80-kube-api-access-6zq4b\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.987813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.987783 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:21.989305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:21.989280 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:22.049301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:22.049265 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zq4b\" (UniqueName: \"kubernetes.io/projected/a76af0b8-8b2e-4be5-821b-c4febd391a80-kube-api-access-6zq4b\") pod \"maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:22.127408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:22.127323 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:22.520347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:22.520316 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w"] Nov 03 19:18:22.521242 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:18:22.521209 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76af0b8_8b2e_4be5_821b_c4febd391a80.slice/crio-ddd412d433e682d55bfd36ed322213df7c08db7bf1c6d3a476f16f2d32168e13 WatchSource:0}: Error finding container ddd412d433e682d55bfd36ed322213df7c08db7bf1c6d3a476f16f2d32168e13: Status 404 returned error can't find the container with id ddd412d433e682d55bfd36ed322213df7c08db7bf1c6d3a476f16f2d32168e13 Nov 03 19:18:22.616049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:22.616002 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" containerID="cri-o://ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63" gracePeriod=45 Nov 03 19:18:22.961495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:22.961467 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:18:23.102510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.102427 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tls-certs\") pod \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " Nov 03 19:18:23.102510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.102465 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-model-cache\") pod \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " Nov 03 19:18:23.102838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.102547 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-dshm\") pod \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " Nov 03 19:18:23.102838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.102635 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46553f9b-0d60-45c3-aabe-2a608c8df7fc-kube-api-access-7gtcb\" (UniqueName: \"kubernetes.io/projected/46553f9b-0d60-45c3-aabe-2a608c8df7fc-kube-api-access-7gtcb\") pod \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " Nov 03 19:18:23.102838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.102774 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tmp-dir\") pod \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " Nov 03 19:18:23.103017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.102863 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-home\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-home\") pod \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\" (UID: \"46553f9b-0d60-45c3-aabe-2a608c8df7fc\") " Nov 03 19:18:23.103017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.102933 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-model-cache" pod "46553f9b-0d60-45c3-aabe-2a608c8df7fc" (UID: "46553f9b-0d60-45c3-aabe-2a608c8df7fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:23.103125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.103077 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tmp-dir" pod "46553f9b-0d60-45c3-aabe-2a608c8df7fc" (UID: "46553f9b-0d60-45c3-aabe-2a608c8df7fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:23.103125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.103111 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-home" pod "46553f9b-0d60-45c3-aabe-2a608c8df7fc" (UID: "46553f9b-0d60-45c3-aabe-2a608c8df7fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:23.103606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.103540 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.103606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.103567 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.103606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.103581 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.105226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.105198 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46553f9b-0d60-45c3-aabe-2a608c8df7fc-kube-api-access-7gtcb" pod "46553f9b-0d60-45c3-aabe-2a608c8df7fc" (UID: "46553f9b-0d60-45c3-aabe-2a608c8df7fc"). InnerVolumeSpecName "kube-api-access-7gtcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:23.105355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.105308 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tls-certs" pod "46553f9b-0d60-45c3-aabe-2a608c8df7fc" (UID: "46553f9b-0d60-45c3-aabe-2a608c8df7fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:23.105579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.105547 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-dshm" pod "46553f9b-0d60-45c3-aabe-2a608c8df7fc" (UID: "46553f9b-0d60-45c3-aabe-2a608c8df7fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:23.204393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.204352 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46553f9b-0d60-45c3-aabe-2a608c8df7fc-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.204393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.204385 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46553f9b-0d60-45c3-aabe-2a608c8df7fc-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.204393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.204395 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gtcb\" (UniqueName: \"kubernetes.io/projected/46553f9b-0d60-45c3-aabe-2a608c8df7fc-kube-api-access-7gtcb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.231426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.231391 4948 generic.go:356] "Generic (PLEG): container finished" podID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerID="ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63" exitCode=0 Nov 03 19:18:23.231632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.231444 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" Nov 03 19:18:23.231632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.231476 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" event={"ID":"46553f9b-0d60-45c3-aabe-2a608c8df7fc","Type":"ContainerDied","Data":"ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63"} Nov 03 19:18:23.231632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.231524 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc" event={"ID":"46553f9b-0d60-45c3-aabe-2a608c8df7fc","Type":"ContainerDied","Data":"8084d4e6296f74780d4faf7e4dd439d2fd75cb3adea67b5f6342648a68ec4ea9"} Nov 03 19:18:23.231632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.231542 4948 scope.go:122] "RemoveContainer" containerID="ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63" Nov 03 19:18:23.233370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.233343 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" event={"ID":"a76af0b8-8b2e-4be5-821b-c4febd391a80","Type":"ContainerStarted","Data":"cfa0e66b2435d8bc57cb27a71f7ee9ce5b63c625d55a64c15898d8c306ee03e4"} Nov 03 19:18:23.233490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.233392 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" event={"ID":"a76af0b8-8b2e-4be5-821b-c4febd391a80","Type":"ContainerStarted","Data":"ddd412d433e682d55bfd36ed322213df7c08db7bf1c6d3a476f16f2d32168e13"} Nov 03 19:18:23.246662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.246638 4948 scope.go:122] "RemoveContainer" containerID="ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63" Nov 03 19:18:23.247022 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:23.246998 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63\": container with ID starting with ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63 not found: ID does not exist" containerID="ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63" Nov 03 19:18:23.247095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.247033 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63"} err="failed to get container status \"ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63\": rpc error: code = NotFound desc = could not find container \"ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63\": container with ID starting with ecfa7231fa0938626fc1e07b5e0a54559111e11d10f7004d070c4c09262e6c63 not found: ID does not exist" Nov 03 19:18:23.403950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.403838 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" podStartSLOduration=2.4038228520000002 podStartE2EDuration="2.403822852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:23.403237859 +0000 UTC m=+2432.886233773" watchObservedRunningTime="2026-11-03 19:18:23.403822852 +0000 UTC m=+2432.886818766" Nov 03 19:18:23.519670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.519630 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc"] Nov 03 19:18:23.528374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.524800 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-w5-3a7056-kserve-6dd597b6c6-998bc"] Nov 03 19:18:23.723197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.723157 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js"] Nov 03 19:18:23.724111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.724085 4948 cpu_manager.go:414] "containerMap: removing container" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" Nov 03 19:18:23.724111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.724112 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" Nov 03 19:18:23.724319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.724311 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" containerName="main" Nov 03 19:18:23.728348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.728323 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.780497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.780458 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js"] Nov 03 19:18:23.814566 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.813037 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.814566 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.813150 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.814566 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.813184 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jcs\" (UniqueName: \"kubernetes.io/projected/a1060ced-d832-4f84-9e91-f52de3558b48-kube-api-access-r5jcs\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.914689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.914647 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.914892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.914740 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.914892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.914774 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jcs\" (UniqueName: \"kubernetes.io/projected/a1060ced-d832-4f84-9e91-f52de3558b48-kube-api-access-r5jcs\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.915473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.915452 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.917215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.917195 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:23.977325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:23.977289 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jcs\" (UniqueName: \"kubernetes.io/projected/a1060ced-d832-4f84-9e91-f52de3558b48-kube-api-access-r5jcs\") pod \"maas-api-revoke-sub-81c07c1f-j29js\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:24.048232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:24.048202 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:24.245146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:24.245057 4948 generic.go:356] "Generic (PLEG): container finished" podID="a76af0b8-8b2e-4be5-821b-c4febd391a80" containerID="cfa0e66b2435d8bc57cb27a71f7ee9ce5b63c625d55a64c15898d8c306ee03e4" exitCode=0 Nov 03 19:18:24.245301 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:24.245172 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" event={"ID":"a76af0b8-8b2e-4be5-821b-c4febd391a80","Type":"ContainerDied","Data":"cfa0e66b2435d8bc57cb27a71f7ee9ce5b63c625d55a64c15898d8c306ee03e4"} Nov 03 19:18:24.480907 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:18:24.480870 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1060ced_d832_4f84_9e91_f52de3558b48.slice/crio-61546884a89967934601a9b5ea00512a488b545ab901b5d50244f5ae8a00d132 WatchSource:0}: Error finding container 61546884a89967934601a9b5ea00512a488b545ab901b5d50244f5ae8a00d132: Status 404 returned error can't find the container with id 61546884a89967934601a9b5ea00512a488b545ab901b5d50244f5ae8a00d132 Nov 03 19:18:24.481759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:24.481730 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js"] Nov 03 19:18:24.756323 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:24.756287 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46553f9b-0d60-45c3-aabe-2a608c8df7fc" path="/var/lib/kubelet/pods/46553f9b-0d60-45c3-aabe-2a608c8df7fc/volumes" Nov 03 19:18:25.255687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.255650 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" event={"ID":"a1060ced-d832-4f84-9e91-f52de3558b48","Type":"ContainerStarted","Data":"64265189dd865cc25674b65606e9682e7dd7066e28a4c0efbabe4e222b9439fc"} Nov 03 19:18:25.256171 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.255742 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" event={"ID":"a1060ced-d832-4f84-9e91-f52de3558b48","Type":"ContainerStarted","Data":"61546884a89967934601a9b5ea00512a488b545ab901b5d50244f5ae8a00d132"} Nov 03 19:18:25.382614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.382554 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" podStartSLOduration=2.382539321 podStartE2EDuration="2.382539321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:25.377835368 +0000 UTC m=+2434.860831283" watchObservedRunningTime="2026-11-03 19:18:25.382539321 +0000 UTC m=+2434.865535274" Nov 03 19:18:25.422917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.422890 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:25.547137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.547054 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-xks-ca\") pod \"a76af0b8-8b2e-4be5-821b-c4febd391a80\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " Nov 03 19:18:25.547324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.547267 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-openshift-ca\") pod \"a76af0b8-8b2e-4be5-821b-c4febd391a80\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " Nov 03 19:18:25.547690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.547657 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-openshift-ca" pod "a76af0b8-8b2e-4be5-821b-c4febd391a80" (UID: "a76af0b8-8b2e-4be5-821b-c4febd391a80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:25.547894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.547872 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76af0b8-8b2e-4be5-821b-c4febd391a80-kube-api-access-6zq4b\" (UniqueName: \"kubernetes.io/projected/a76af0b8-8b2e-4be5-821b-c4febd391a80-kube-api-access-6zq4b\") pod \"a76af0b8-8b2e-4be5-821b-c4febd391a80\" (UID: \"a76af0b8-8b2e-4be5-821b-c4febd391a80\") " Nov 03 19:18:25.548461 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.548431 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:25.550118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.550091 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-xks-ca" pod "a76af0b8-8b2e-4be5-821b-c4febd391a80" (UID: "a76af0b8-8b2e-4be5-821b-c4febd391a80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:25.552177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.552144 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76af0b8-8b2e-4be5-821b-c4febd391a80-kube-api-access-6zq4b" pod "a76af0b8-8b2e-4be5-821b-c4febd391a80" (UID: "a76af0b8-8b2e-4be5-821b-c4febd391a80"). InnerVolumeSpecName "kube-api-access-6zq4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:25.649469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.649428 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zq4b\" (UniqueName: \"kubernetes.io/projected/a76af0b8-8b2e-4be5-821b-c4febd391a80-kube-api-access-6zq4b\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:25.649469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:25.649461 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76af0b8-8b2e-4be5-821b-c4febd391a80-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:26.270598 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.270568 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" Nov 03 19:18:26.271100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.270611 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w" event={"ID":"a76af0b8-8b2e-4be5-821b-c4febd391a80","Type":"ContainerDied","Data":"ddd412d433e682d55bfd36ed322213df7c08db7bf1c6d3a476f16f2d32168e13"} Nov 03 19:18:26.271100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.270665 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddd412d433e682d55bfd36ed322213df7c08db7bf1c6d3a476f16f2d32168e13" Nov 03 19:18:26.273349 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.273305 4948 generic.go:356] "Generic (PLEG): container finished" podID="a1060ced-d832-4f84-9e91-f52de3558b48" containerID="64265189dd865cc25674b65606e9682e7dd7066e28a4c0efbabe4e222b9439fc" exitCode=0 Nov 03 19:18:26.273519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.273407 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" event={"ID":"a1060ced-d832-4f84-9e91-f52de3558b48","Type":"ContainerDied","Data":"64265189dd865cc25674b65606e9682e7dd7066e28a4c0efbabe4e222b9439fc"} Nov 03 19:18:26.278648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.278622 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn"] Nov 03 19:18:26.278892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.278869 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" containerID="cri-o://d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d" gracePeriod=30 Nov 03 19:18:26.704479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:26.704438 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" podUID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:27.486446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.486410 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:27.582996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.582907 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-openshift-ca\") pod \"a1060ced-d832-4f84-9e91-f52de3558b48\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " Nov 03 19:18:27.582996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.582951 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1060ced-d832-4f84-9e91-f52de3558b48-kube-api-access-r5jcs\" (UniqueName: \"kubernetes.io/projected/a1060ced-d832-4f84-9e91-f52de3558b48-kube-api-access-r5jcs\") pod \"a1060ced-d832-4f84-9e91-f52de3558b48\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " Nov 03 19:18:27.583250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.583067 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-xks-ca\") pod \"a1060ced-d832-4f84-9e91-f52de3558b48\" (UID: \"a1060ced-d832-4f84-9e91-f52de3558b48\") " Nov 03 19:18:27.583371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.583332 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-openshift-ca" pod "a1060ced-d832-4f84-9e91-f52de3558b48" (UID: "a1060ced-d832-4f84-9e91-f52de3558b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:27.583653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.583632 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:27.585742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.585715 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1060ced-d832-4f84-9e91-f52de3558b48-kube-api-access-r5jcs" pod "a1060ced-d832-4f84-9e91-f52de3558b48" (UID: "a1060ced-d832-4f84-9e91-f52de3558b48"). InnerVolumeSpecName "kube-api-access-r5jcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:27.585742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.585725 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-xks-ca" pod "a1060ced-d832-4f84-9e91-f52de3558b48" (UID: "a1060ced-d832-4f84-9e91-f52de3558b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:27.684948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.684916 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1060ced-d832-4f84-9e91-f52de3558b48-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:27.684948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:27.684945 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5jcs\" (UniqueName: \"kubernetes.io/projected/a1060ced-d832-4f84-9e91-f52de3558b48-kube-api-access-r5jcs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:28.291965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:28.291913 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" event={"ID":"a1060ced-d832-4f84-9e91-f52de3558b48","Type":"ContainerDied","Data":"61546884a89967934601a9b5ea00512a488b545ab901b5d50244f5ae8a00d132"} Nov 03 19:18:28.292176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:28.291996 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61546884a89967934601a9b5ea00512a488b545ab901b5d50244f5ae8a00d132" Nov 03 19:18:28.292290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:28.292263 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js" Nov 03 19:18:31.573657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.573628 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:18:31.647093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647056 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-token\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647102 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4f8681-850f-4ce3-baa4-368173ab3179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f8681-850f-4ce3-baa4-368173ab3179-istiod-ca-cert\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647162 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-certs\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647212 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-data\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647322 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-podinfo\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647385 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-socket\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647439 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-kube-api-access-kjf7n\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-kube-api-access-kjf7n\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647478 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-envoy\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.647633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647518 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-data" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:31.647633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647533 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-certs" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:31.647633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647624 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4f8681-850f-4ce3-baa4-368173ab3179-istiod-ca-cert" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:31.647825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.647632 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-credential-socket\") pod \"ee4f8681-850f-4ce3-baa4-368173ab3179\" (UID: \"ee4f8681-850f-4ce3-baa4-368173ab3179\") " Nov 03 19:18:31.648026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.648000 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-credential-socket" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:31.648167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.648047 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-socket" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:31.648295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.648273 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.648342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.648303 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f8681-850f-4ce3-baa4-368173ab3179-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.648342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.648319 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.648342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.648332 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.648438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.648347 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.649552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.649520 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-token" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:31.649678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.649657 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-envoy" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:31.649901 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.649882 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-podinfo" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:18:31.649967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.649911 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-kube-api-access-kjf7n" pod "ee4f8681-850f-4ce3-baa4-368173ab3179" (UID: "ee4f8681-850f-4ce3-baa4-368173ab3179"). InnerVolumeSpecName "kube-api-access-kjf7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:31.749431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.749395 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.749431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.749431 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjf7n\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-kube-api-access-kjf7n\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.749631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.749443 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.749631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.749454 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f8681-850f-4ce3-baa4-368173ab3179-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:31.812767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.812653 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw"] Nov 03 19:18:31.819044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:31.819012 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg"] Nov 03 19:18:32.347046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.347014 4948 generic.go:356] "Generic (PLEG): container finished" podID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerID="d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d" exitCode=0 Nov 03 19:18:32.347276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.347082 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" Nov 03 19:18:32.347276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.347100 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" event={"ID":"ee4f8681-850f-4ce3-baa4-368173ab3179","Type":"ContainerDied","Data":"d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d"} Nov 03 19:18:32.347276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.347197 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn" event={"ID":"ee4f8681-850f-4ce3-baa4-368173ab3179","Type":"ContainerDied","Data":"0538da0abbad9233f3c3c5dd05798ffb31d3af623fa8b47df1bb437520ff73fa"} Nov 03 19:18:32.347276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.347229 4948 scope.go:122] "RemoveContainer" containerID="d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d" Nov 03 19:18:32.360509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.360487 4948 scope.go:122] "RemoveContainer" containerID="c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762" Nov 03 19:18:32.371475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.371449 4948 scope.go:122] "RemoveContainer" containerID="d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d" Nov 03 19:18:32.371836 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:32.371810 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d\": container with ID starting with d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d not found: ID does not exist" containerID="d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d" Nov 03 19:18:32.371910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.371850 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d"} err="failed to get container status \"d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d\": rpc error: code = NotFound desc = could not find container \"d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d\": container with ID starting with d28e238c14dc5fdecc50d1dbb2e67a55c96f706f9f173776dcd7e36d6583a77d not found: ID does not exist" Nov 03 19:18:32.371910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.371877 4948 scope.go:122] "RemoveContainer" containerID="c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762" Nov 03 19:18:32.372215 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:32.372195 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762\": container with ID starting with c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762 not found: ID does not exist" containerID="c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762" Nov 03 19:18:32.372264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.372221 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762"} err="failed to get container status \"c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762\": rpc error: code = NotFound desc = could not find container \"c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762\": container with ID starting with c0c42fabfcb6f8806c0caa690d6b7f52d432c93bfb1edc72e77ca474360e6762 not found: ID does not exist" Nov 03 19:18:32.521426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.521394 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn"] Nov 03 19:18:32.573712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.573668 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-efd81c-openshift-default-7956bf8cfc-cfjrn"] Nov 03 19:18:32.756568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:32.756532 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" path="/var/lib/kubelet/pods/ee4f8681-850f-4ce3-baa4-368173ab3179/volumes" Nov 03 19:18:32.797569 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:32.797523 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:18:33.484671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:33.484260 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8"] Nov 03 19:18:36.704597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:36.704551 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" podUID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:38.486736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:38.485781 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" podUID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" containerName="payload-pre-processing" containerID="cri-o://5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238" gracePeriod=25 Nov 03 19:18:38.880935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:38.880832 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7"] Nov 03 19:18:39.012947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.012905 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57"] Nov 03 19:18:39.385564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.385532 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 19:18:39.574795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.567578 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-kube-api-access-f42l2\" (UniqueName: \"kubernetes.io/projected/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-kube-api-access-f42l2\") pod \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " Nov 03 19:18:39.574795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.567830 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-plugins-config-volume\") pod \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\" (UID: \"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2\") " Nov 03 19:18:39.670722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.578819 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-plugins-config-volume" pod "2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" (UID: "2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:39.670722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.641830 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-kube-api-access-f42l2" pod "2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" (UID: "2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2"). InnerVolumeSpecName "kube-api-access-f42l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:39.722980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.722569 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f42l2\" (UniqueName: \"kubernetes.io/projected/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-kube-api-access-f42l2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:39.722980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.722609 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:39.815113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.771346 4948 generic.go:356] "Generic (PLEG): container finished" podID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" containerID="7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d" exitCode=0 Nov 03 19:18:39.815113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.771573 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" event={"ID":"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2","Type":"ContainerDied","Data":"7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d"} Nov 03 19:18:39.815113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.771625 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" event={"ID":"2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2","Type":"ContainerDied","Data":"a5d874760e02652e534a61848cc7bcdaf791f78a1dc253981a8ce27385209a28"} Nov 03 19:18:39.815113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.771654 4948 scope.go:122] "RemoveContainer" containerID="7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d" Nov 03 19:18:39.815113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:39.771814 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp" Nov 03 19:18:40.016112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:40.016084 4948 scope.go:122] "RemoveContainer" containerID="7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d" Nov 03 19:18:40.023723 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:40.020215 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d\": container with ID starting with 7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d not found: ID does not exist" containerID="7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d" Nov 03 19:18:40.023723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:40.020264 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d"} err="failed to get container status \"7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d\": rpc error: code = NotFound desc = could not find container \"7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d\": container with ID starting with 7ca87559032a23db93c4b437cfa2510d19d8cf2060de7f376abc23e98ab9aa9d not found: ID does not exist" Nov 03 19:18:40.023723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:40.020345 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp"] Nov 03 19:18:40.460318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:40.450218 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3a7056-7bc4d594f7-wklrp"] Nov 03 19:18:40.673831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:40.668391 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" podUID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:40.956119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:40.956074 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" path="/var/lib/kubelet/pods/2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2/volumes" Nov 03 19:18:41.342908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:41.342804 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:18:43.885736 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:43.884821 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" podUID="98a3e420-71c3-4cca-9036-323dfd7f1eba" containerName="payload-processing" containerID="cri-o://1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d" gracePeriod=25 Nov 03 19:18:44.014181 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:44.014138 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" podUID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" containerName="payload-pre-processing" containerID="cri-o://9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305" gracePeriod=25 Nov 03 19:18:46.834852 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:46.834803 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" containerID="cri-o://1617fda5cdf9e5194bd8a78ad342305816f0e1fcee3824da849e7f3622c9fa44" gracePeriod=45 Nov 03 19:18:46.840543 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:46.840307 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" containerID="cri-o://279ba8cf87e577c77c219a583eb804205828a18b6f5e851176f0a7a24d64d26a" gracePeriod=45 Nov 03 19:18:47.154511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.154467 4948 generic.go:356] "Generic (PLEG): container finished" podID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerID="1617fda5cdf9e5194bd8a78ad342305816f0e1fcee3824da849e7f3622c9fa44" exitCode=0 Nov 03 19:18:47.154716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.154599 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" event={"ID":"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c","Type":"ContainerDied","Data":"1617fda5cdf9e5194bd8a78ad342305816f0e1fcee3824da849e7f3622c9fa44"} Nov 03 19:18:47.154716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.154640 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" event={"ID":"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c","Type":"ContainerDied","Data":"599ffcbc66c987bfb12dec250e9a7de9f983f1e4f7a9b8478279f24319f438b1"} Nov 03 19:18:47.154716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.154658 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="599ffcbc66c987bfb12dec250e9a7de9f983f1e4f7a9b8478279f24319f438b1" Nov 03 19:18:47.156871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.156451 4948 generic.go:356] "Generic (PLEG): container finished" podID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerID="279ba8cf87e577c77c219a583eb804205828a18b6f5e851176f0a7a24d64d26a" exitCode=0 Nov 03 19:18:47.156871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.156761 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" event={"ID":"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec","Type":"ContainerDied","Data":"279ba8cf87e577c77c219a583eb804205828a18b6f5e851176f0a7a24d64d26a"} Nov 03 19:18:47.156871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.156795 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" event={"ID":"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec","Type":"ContainerDied","Data":"5f09b21ad1838476ccde9f939bc6c96ffb9681518df065cd7045f3701b9bcf32"} Nov 03 19:18:47.156871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.156830 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f09b21ad1838476ccde9f939bc6c96ffb9681518df065cd7045f3701b9bcf32" Nov 03 19:18:47.172386 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.172355 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:18:47.176580 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.176548 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:18:47.246824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.246791 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-home\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-home\") pod \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " Nov 03 19:18:47.247071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.246838 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-dshm\") pod \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " Nov 03 19:18:47.247071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.246901 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-kube-api-access-t2j24\" (UniqueName: \"kubernetes.io/projected/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-kube-api-access-t2j24\") pod \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " Nov 03 19:18:47.247071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.246959 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-model-cache\") pod \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " Nov 03 19:18:47.247071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.246980 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tls-certs\") pod \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " Nov 03 19:18:47.247071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247037 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-kube-api-access-xg8d2\" (UniqueName: \"kubernetes.io/projected/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-kube-api-access-xg8d2\") pod \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " Nov 03 19:18:47.247071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247060 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tmp-dir\") pod \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " Nov 03 19:18:47.247071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247077 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tls-certs\") pod \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " Nov 03 19:18:47.247716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247094 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-model-cache\") pod \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " Nov 03 19:18:47.247716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247113 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-dshm\") pod \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " Nov 03 19:18:47.247716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247134 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-home\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-home\") pod \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\" (UID: \"9d3f7d8c-ba02-44e3-90fe-97c5c8206bec\") " Nov 03 19:18:47.247716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247173 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tmp-dir\") pod \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\" (UID: \"3f6d0ca0-ce58-4983-bc97-8222e54b8c1c\") " Nov 03 19:18:47.248028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.247882 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tmp-dir" pod "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" (UID: "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.248094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.248077 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-home" pod "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" (UID: "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.248618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.248588 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tmp-dir" pod "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" (UID: "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.250008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.249749 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-model-cache" pod "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" (UID: "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.250179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.250132 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-model-cache" pod "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" (UID: "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.250343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.250313 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-home" pod "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" (UID: "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.256145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.256086 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-kube-api-access-t2j24" pod "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" (UID: "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec"). InnerVolumeSpecName "kube-api-access-t2j24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:47.261232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.261193 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-dshm" pod "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" (UID: "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.261347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.261216 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tls-certs" pod "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" (UID: "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:47.262121 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.262085 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-kube-api-access-xg8d2" pod "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" (UID: "3f6d0ca0-ce58-4983-bc97-8222e54b8c1c"). InnerVolumeSpecName "kube-api-access-xg8d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:47.262336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.262302 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-dshm" pod "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" (UID: "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.272914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.272366 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tls-certs" pod "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" (UID: "9d3f7d8c-ba02-44e3-90fe-97c5c8206bec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:47.348983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.348946 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.348983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.348977 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.348983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.348987 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg8d2\" (UniqueName: \"kubernetes.io/projected/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-kube-api-access-xg8d2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.348996 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349005 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349014 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349022 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349030 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349038 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349045 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349052 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.349318 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:47.349060 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2j24\" (UniqueName: \"kubernetes.io/projected/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec-kube-api-access-t2j24\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:48.166891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.166806 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw" Nov 03 19:18:48.167348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.166823 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg" Nov 03 19:18:48.341052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.341013 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg"] Nov 03 19:18:48.389467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.389431 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-w3-a4cacd-kserve-5b4d546576-nnljg"] Nov 03 19:18:48.455034 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.454986 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw"] Nov 03 19:18:48.461352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.461317 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/distinct-2-w3-a4cacd-kserve-bc94cb899-4xltw"] Nov 03 19:18:48.756980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.756894 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" path="/var/lib/kubelet/pods/3f6d0ca0-ce58-4983-bc97-8222e54b8c1c/volumes" Nov 03 19:18:48.757518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:48.757491 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" path="/var/lib/kubelet/pods/9d3f7d8c-ba02-44e3-90fe-97c5c8206bec/volumes" Nov 03 19:18:49.486353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.486314 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" podUID="98a3e420-71c3-4cca-9036-323dfd7f1eba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:49.486353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.486328 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" podUID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:49.493791 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.493762 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2"] Nov 03 19:18:49.494512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494491 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494513 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494529 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" containerName="payload-processing" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494536 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" containerName="payload-processing" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494549 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494557 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494609 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494617 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494627 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a76af0b8-8b2e-4be5-821b-c4febd391a80" containerName="revoke-keys" Nov 03 19:18:49.494633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494634 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76af0b8-8b2e-4be5-821b-c4febd391a80" containerName="revoke-keys" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494663 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a1060ced-d832-4f84-9e91-f52de3558b48" containerName="revoke-keys" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494671 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1060ced-d832-4f84-9e91-f52de3558b48" containerName="revoke-keys" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494858 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1060ced-d832-4f84-9e91-f52de3558b48" containerName="revoke-keys" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494878 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494890 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed68fe7-17e0-4678-bbe3-caf1e5d46ac2" containerName="payload-processing" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494902 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6d0ca0-ce58-4983-bc97-8222e54b8c1c" containerName="main" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494915 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494930 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3f7d8c-ba02-44e3-90fe-97c5c8206bec" containerName="main" Nov 03 19:18:49.495116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.494943 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76af0b8-8b2e-4be5-821b-c4febd391a80" containerName="revoke-keys" Nov 03 19:18:49.498007 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.497986 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.590803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.590670 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.590985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.590968 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.591061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.591006 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wf4q\" (UniqueName: \"kubernetes.io/projected/b7d11faf-4101-49c8-a416-d60bd5147fe3-kube-api-access-5wf4q\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.596024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.595970 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:18:49.596024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.595974 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:18:49.609468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.609438 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2"] Nov 03 19:18:49.692135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.692088 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.692135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.692137 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wf4q\" (UniqueName: \"kubernetes.io/projected/b7d11faf-4101-49c8-a416-d60bd5147fe3-kube-api-access-5wf4q\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.692403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.692237 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.692930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.692901 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.694875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.694849 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.754384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.754344 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wf4q\" (UniqueName: \"kubernetes.io/projected/b7d11faf-4101-49c8-a416-d60bd5147fe3-kube-api-access-5wf4q\") pod \"maas-api-revoke-sub-69d4b55a-5t5b2\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:49.811575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:49.811537 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:50.188879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.188840 4948 generic.go:356] "Generic (PLEG): container finished" podID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerID="49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e" exitCode=0 Nov 03 19:18:50.189065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.188926 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" event={"ID":"42791c9a-d4ed-4055-b330-f565ef971dc7","Type":"ContainerDied","Data":"49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e"} Nov 03 19:18:50.189432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.189414 4948 scope.go:122] "RemoveContainer" containerID="49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e" Nov 03 19:18:50.191325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.191290 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:18:50.191465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.191382 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:18:50.191465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.191442 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:18:50.198040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.198016 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2"] Nov 03 19:18:50.198359 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:18:50.198335 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d11faf_4101_49c8_a416_d60bd5147fe3.slice/crio-d3aa975c5538c06bf5f21b49f33a1b1e549262206753507aa96efd9e4f7c3932 WatchSource:0}: Error finding container d3aa975c5538c06bf5f21b49f33a1b1e549262206753507aa96efd9e4f7c3932: Status 404 returned error can't find the container with id d3aa975c5538c06bf5f21b49f33a1b1e549262206753507aa96efd9e4f7c3932 Nov 03 19:18:50.591254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:50.591213 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" podUID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:51.199822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:51.199786 4948 generic.go:356] "Generic (PLEG): container finished" podID="b7d11faf-4101-49c8-a416-d60bd5147fe3" containerID="69585d98fb69d13f48064197385e009306a94a4140483f492fb1372f57aa295b" exitCode=0 Nov 03 19:18:51.200044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:51.199886 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" event={"ID":"b7d11faf-4101-49c8-a416-d60bd5147fe3","Type":"ContainerDied","Data":"69585d98fb69d13f48064197385e009306a94a4140483f492fb1372f57aa295b"} Nov 03 19:18:51.200044 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:51.199956 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" event={"ID":"b7d11faf-4101-49c8-a416-d60bd5147fe3","Type":"ContainerStarted","Data":"d3aa975c5538c06bf5f21b49f33a1b1e549262206753507aa96efd9e4f7c3932"} Nov 03 19:18:51.202452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:51.202427 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" event={"ID":"42791c9a-d4ed-4055-b330-f565ef971dc7","Type":"ContainerStarted","Data":"bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad"} Nov 03 19:18:51.781724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:51.780491 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:18:51.781724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:51.780620 4948 patch_prober.go:29] interesting pod/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 19:18:51.781724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:51.780675 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 19:18:52.424208 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.424179 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:52.527660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.527573 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d11faf-4101-49c8-a416-d60bd5147fe3-kube-api-access-5wf4q\" (UniqueName: \"kubernetes.io/projected/b7d11faf-4101-49c8-a416-d60bd5147fe3-kube-api-access-5wf4q\") pod \"b7d11faf-4101-49c8-a416-d60bd5147fe3\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " Nov 03 19:18:52.527660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.527642 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-xks-ca\") pod \"b7d11faf-4101-49c8-a416-d60bd5147fe3\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " Nov 03 19:18:52.527952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.527729 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-openshift-ca\") pod \"b7d11faf-4101-49c8-a416-d60bd5147fe3\" (UID: \"b7d11faf-4101-49c8-a416-d60bd5147fe3\") " Nov 03 19:18:52.528158 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.528133 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-openshift-ca" pod "b7d11faf-4101-49c8-a416-d60bd5147fe3" (UID: "b7d11faf-4101-49c8-a416-d60bd5147fe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:52.528260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.528241 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:52.529785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.529756 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-xks-ca" pod "b7d11faf-4101-49c8-a416-d60bd5147fe3" (UID: "b7d11faf-4101-49c8-a416-d60bd5147fe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:52.529898 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.529854 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d11faf-4101-49c8-a416-d60bd5147fe3-kube-api-access-5wf4q" pod "b7d11faf-4101-49c8-a416-d60bd5147fe3" (UID: "b7d11faf-4101-49c8-a416-d60bd5147fe3"). InnerVolumeSpecName "kube-api-access-5wf4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:52.630084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.630046 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wf4q\" (UniqueName: \"kubernetes.io/projected/b7d11faf-4101-49c8-a416-d60bd5147fe3-kube-api-access-5wf4q\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:52.630084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.630084 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d11faf-4101-49c8-a416-d60bd5147fe3-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:52.794892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.786076 4948 patch_prober.go:29] interesting pod/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 19:18:52.794892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:52.786159 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 19:18:53.221794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:53.221753 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" Nov 03 19:18:53.221978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:53.221823 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2" event={"ID":"b7d11faf-4101-49c8-a416-d60bd5147fe3","Type":"ContainerDied","Data":"d3aa975c5538c06bf5f21b49f33a1b1e549262206753507aa96efd9e4f7c3932"} Nov 03 19:18:53.221978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:53.221884 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3aa975c5538c06bf5f21b49f33a1b1e549262206753507aa96efd9e4f7c3932" Nov 03 19:18:53.784722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:53.784185 4948 patch_prober.go:29] interesting pod/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 19:18:53.784722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:53.784264 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 19:18:54.788726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:54.783143 4948 patch_prober.go:29] interesting pod/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 19:18:54.788726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:54.783219 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 19:18:55.047868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.047783 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn"] Nov 03 19:18:55.048459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.048445 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" Nov 03 19:18:55.048515 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.048460 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4f8681-850f-4ce3-baa4-368173ab3179" containerName="istio-proxy" Nov 03 19:18:55.048515 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.048501 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b7d11faf-4101-49c8-a416-d60bd5147fe3" containerName="revoke-keys" Nov 03 19:18:55.048515 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.048506 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d11faf-4101-49c8-a416-d60bd5147fe3" containerName="revoke-keys" Nov 03 19:18:55.048623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.048613 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d11faf-4101-49c8-a416-d60bd5147fe3" containerName="revoke-keys" Nov 03 19:18:55.068724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.051171 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.105965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.105109 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-942af8-serving-cert\"" Nov 03 19:18:55.135398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.135287 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn"] Nov 03 19:18:55.196759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.194330 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.196759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.194464 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwsml\" (UniqueName: \"kubernetes.io/projected/cdabd8f8-e278-4673-bebf-10c6d30259ab-kube-api-access-qwsml\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.196759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.194504 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdabd8f8-e278-4673-bebf-10c6d30259ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.196759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.194619 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-metrics-certs\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.329226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.323183 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-metrics-certs\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.329226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.323256 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.329226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.323434 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwsml\" (UniqueName: \"kubernetes.io/projected/cdabd8f8-e278-4673-bebf-10c6d30259ab-kube-api-access-qwsml\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.329226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.323490 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdabd8f8-e278-4673-bebf-10c6d30259ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.329226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.324427 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdabd8f8-e278-4673-bebf-10c6d30259ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.334608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.334510 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.335434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.335114 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-metrics-certs\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.387752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.387215 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwsml\" (UniqueName: \"kubernetes.io/projected/cdabd8f8-e278-4673-bebf-10c6d30259ab-kube-api-access-qwsml\") pod \"maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.396846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.396816 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:55.783680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.783489 4948 patch_prober.go:29] interesting pod/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 19:18:55.783680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.783578 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 19:18:55.930020 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:18:55.929991 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdabd8f8_e278_4673_bebf_10c6d30259ab.slice/crio-91108ea27befcda785c8588518bdf25c7247eb03aa16df521075a85b43b54cfc WatchSource:0}: Error finding container 91108ea27befcda785c8588518bdf25c7247eb03aa16df521075a85b43b54cfc: Status 404 returned error can't find the container with id 91108ea27befcda785c8588518bdf25c7247eb03aa16df521075a85b43b54cfc Nov 03 19:18:55.930465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:55.930367 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn"] Nov 03 19:18:56.250221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:56.250179 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" event={"ID":"cdabd8f8-e278-4673-bebf-10c6d30259ab","Type":"ContainerStarted","Data":"91108ea27befcda785c8588518bdf25c7247eb03aa16df521075a85b43b54cfc"} Nov 03 19:18:56.807876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:56.807840 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:18:56.809000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:56.808973 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:18:56.812750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:56.810207 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:18:57.259835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:57.259797 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" event={"ID":"cdabd8f8-e278-4673-bebf-10c6d30259ab","Type":"ContainerStarted","Data":"6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f"} Nov 03 19:18:57.260316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:57.259865 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:18:57.384659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:57.384603 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" podStartSLOduration=2.52739051 podStartE2EDuration="3.384587884s" totalImagesPullingTime="857.197374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:57.381832935 +0000 UTC m=+2466.864828862" watchObservedRunningTime="2026-11-03 19:18:57.384587884 +0000 UTC m=+2466.867583797" Nov 03 19:18:58.790027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:58.789965 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 19:18:58.870066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:58.870032 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-kube-api-access-dvqqn\" (UniqueName: \"kubernetes.io/projected/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-kube-api-access-dvqqn\") pod \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " Nov 03 19:18:58.870066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:58.870069 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-plugins-config-volume\") pod \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\" (UID: \"1fc2b876-e5c6-4008-9558-f3568dc1a6e8\") " Nov 03 19:18:58.870590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:58.870553 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-plugins-config-volume" pod "1fc2b876-e5c6-4008-9558-f3568dc1a6e8" (UID: "1fc2b876-e5c6-4008-9558-f3568dc1a6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:58.872172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:58.872148 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-kube-api-access-dvqqn" pod "1fc2b876-e5c6-4008-9558-f3568dc1a6e8" (UID: "1fc2b876-e5c6-4008-9558-f3568dc1a6e8"). InnerVolumeSpecName "kube-api-access-dvqqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:58.971935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:58.971894 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqqn\" (UniqueName: \"kubernetes.io/projected/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-kube-api-access-dvqqn\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:58.971935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:58.971925 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc2b876-e5c6-4008-9558-f3568dc1a6e8-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:59.283170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.283133 4948 generic.go:356] "Generic (PLEG): container finished" podID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" containerID="5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238" exitCode=0 Nov 03 19:18:59.283324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.283186 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" Nov 03 19:18:59.283324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.283217 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" event={"ID":"1fc2b876-e5c6-4008-9558-f3568dc1a6e8","Type":"ContainerDied","Data":"5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238"} Nov 03 19:18:59.283324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.283284 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8" event={"ID":"1fc2b876-e5c6-4008-9558-f3568dc1a6e8","Type":"ContainerDied","Data":"c738bcd0f58d985a69654b05e31c2a8ea7cbfc42bbf00a8e85005d295e3d0474"} Nov 03 19:18:59.283324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.283317 4948 scope.go:122] "RemoveContainer" containerID="5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238" Nov 03 19:18:59.295295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.295275 4948 scope.go:122] "RemoveContainer" containerID="5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238" Nov 03 19:18:59.295587 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:18:59.295567 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238\": container with ID starting with 5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238 not found: ID does not exist" containerID="5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238" Nov 03 19:18:59.295645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.295596 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238"} err="failed to get container status \"5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238\": rpc error: code = NotFound desc = could not find container \"5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238\": container with ID starting with 5ddc709b32ddec256fafdc90b5069835ef7ca6bedbec64c794cdb40dbf758238 not found: ID does not exist" Nov 03 19:18:59.452830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.452790 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8"] Nov 03 19:18:59.485751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.485693 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" podUID="98a3e420-71c3-4cca-9036-323dfd7f1eba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:59.485936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.485801 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" podUID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:18:59.504014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:18:59.503983 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9686c3-6f6bdb5d59-twjx8"] Nov 03 19:19:00.250383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:00.250346 4948 scope.go:122] "RemoveContainer" containerID="1617fda5cdf9e5194bd8a78ad342305816f0e1fcee3824da849e7f3622c9fa44" Nov 03 19:19:00.261584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:00.261561 4948 scope.go:122] "RemoveContainer" containerID="279ba8cf87e577c77c219a583eb804205828a18b6f5e851176f0a7a24d64d26a" Nov 03 19:19:00.273147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:00.273099 4948 scope.go:122] "RemoveContainer" containerID="73fb8443a70ed3117408bbb82babdda1bac22f02673ebd2ebcd0e0ffc197ea11" Nov 03 19:19:00.757017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:00.756915 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" path="/var/lib/kubelet/pods/1fc2b876-e5c6-4008-9558-f3568dc1a6e8/volumes" Nov 03 19:19:02.795173 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:02.795131 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:19:03.279893 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:03.279857 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:19:04.173935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.173836 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 19:19:04.250139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.250108 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a3e420-71c3-4cca-9036-323dfd7f1eba-kube-api-access-5l49x\" (UniqueName: \"kubernetes.io/projected/98a3e420-71c3-4cca-9036-323dfd7f1eba-kube-api-access-5l49x\") pod \"98a3e420-71c3-4cca-9036-323dfd7f1eba\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " Nov 03 19:19:04.250338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.250312 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a3e420-71c3-4cca-9036-323dfd7f1eba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a3e420-71c3-4cca-9036-323dfd7f1eba-plugins-config-volume\") pod \"98a3e420-71c3-4cca-9036-323dfd7f1eba\" (UID: \"98a3e420-71c3-4cca-9036-323dfd7f1eba\") " Nov 03 19:19:04.250821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.250789 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a3e420-71c3-4cca-9036-323dfd7f1eba-plugins-config-volume" pod "98a3e420-71c3-4cca-9036-323dfd7f1eba" (UID: "98a3e420-71c3-4cca-9036-323dfd7f1eba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:04.253405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.253377 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a3e420-71c3-4cca-9036-323dfd7f1eba-kube-api-access-5l49x" pod "98a3e420-71c3-4cca-9036-323dfd7f1eba" (UID: "98a3e420-71c3-4cca-9036-323dfd7f1eba"). InnerVolumeSpecName "kube-api-access-5l49x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:04.286612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.286590 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 19:19:04.331310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.331270 4948 generic.go:356] "Generic (PLEG): container finished" podID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" containerID="9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305" exitCode=0 Nov 03 19:19:04.331618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.331375 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" Nov 03 19:19:04.331618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.331459 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" event={"ID":"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4","Type":"ContainerDied","Data":"9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305"} Nov 03 19:19:04.331618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.331512 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57" event={"ID":"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4","Type":"ContainerDied","Data":"6c25e38641426f7dc3da7091604088457fd2d6a3e57413cfc4461d89088b540b"} Nov 03 19:19:04.331618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.331544 4948 scope.go:122] "RemoveContainer" containerID="9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305" Nov 03 19:19:04.334250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.334222 4948 generic.go:356] "Generic (PLEG): container finished" podID="98a3e420-71c3-4cca-9036-323dfd7f1eba" containerID="1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d" exitCode=0 Nov 03 19:19:04.334406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.334304 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" Nov 03 19:19:04.334406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.334388 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" event={"ID":"98a3e420-71c3-4cca-9036-323dfd7f1eba","Type":"ContainerDied","Data":"1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d"} Nov 03 19:19:04.334590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.334428 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7" event={"ID":"98a3e420-71c3-4cca-9036-323dfd7f1eba","Type":"ContainerDied","Data":"5ef9b3adeaba74492f933a8c191a206f9f117cefda45f587b5295b7f891017cb"} Nov 03 19:19:04.346955 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.346939 4948 scope.go:122] "RemoveContainer" containerID="9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305" Nov 03 19:19:04.347292 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:04.347270 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305\": container with ID starting with 9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305 not found: ID does not exist" containerID="9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305" Nov 03 19:19:04.347362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.347300 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305"} err="failed to get container status \"9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305\": rpc error: code = NotFound desc = could not find container \"9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305\": container with ID starting with 9be0f897898576b7ca9ab730763863d3816a26e8c7f880423c4044ba75509305 not found: ID does not exist" Nov 03 19:19:04.347362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.347319 4948 scope.go:122] "RemoveContainer" containerID="1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d" Nov 03 19:19:04.351639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.351599 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-kube-api-access-t9mdq\" (UniqueName: \"kubernetes.io/projected/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-kube-api-access-t9mdq\") pod \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " Nov 03 19:19:04.351824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.351808 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-plugins-config-volume\") pod \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\" (UID: \"bbd2d734-ddb8-47d5-8a3d-c149f684b6d4\") " Nov 03 19:19:04.352159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.352140 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-plugins-config-volume" pod "bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" (UID: "bbd2d734-ddb8-47d5-8a3d-c149f684b6d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:04.352308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.352294 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:04.352355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.352310 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l49x\" (UniqueName: \"kubernetes.io/projected/98a3e420-71c3-4cca-9036-323dfd7f1eba-kube-api-access-5l49x\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:04.352355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.352320 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a3e420-71c3-4cca-9036-323dfd7f1eba-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:04.353617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.353601 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-kube-api-access-t9mdq" pod "bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" (UID: "bbd2d734-ddb8-47d5-8a3d-c149f684b6d4"). InnerVolumeSpecName "kube-api-access-t9mdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:04.356831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.356815 4948 scope.go:122] "RemoveContainer" containerID="1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d" Nov 03 19:19:04.357126 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:04.357109 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d\": container with ID starting with 1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d not found: ID does not exist" containerID="1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d" Nov 03 19:19:04.357172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.357133 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d"} err="failed to get container status \"1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d\": rpc error: code = NotFound desc = could not find container \"1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d\": container with ID starting with 1f7af7492b2bfc9b1abee957fbaee5ad1d9da0d83afc3936fea4574df4deea1d not found: ID does not exist" Nov 03 19:19:04.453639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.453590 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9mdq\" (UniqueName: \"kubernetes.io/projected/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4-kube-api-access-t9mdq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:04.579368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.579333 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7"] Nov 03 19:19:04.583067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.583042 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a4cacd-656dcc8dd5-2wst7"] Nov 03 19:19:04.756098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.755999 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a3e420-71c3-4cca-9036-323dfd7f1eba" path="/var/lib/kubelet/pods/98a3e420-71c3-4cca-9036-323dfd7f1eba/volumes" Nov 03 19:19:04.759635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.759606 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57"] Nov 03 19:19:04.763418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:04.763391 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a4cacd-57c5b47ff9-kfm57"] Nov 03 19:19:06.754665 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:06.754631 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" path="/var/lib/kubelet/pods/bbd2d734-ddb8-47d5-8a3d-c149f684b6d4/volumes" Nov 03 19:19:10.952356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:10.952309 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw"] Nov 03 19:19:11.002929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:11.002891 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-69e2b9-zqvvw"] Nov 03 19:19:12.755421 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:12.755382 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c51611c-8f6f-4fc3-988c-93b1a25427a2" path="/var/lib/kubelet/pods/9c51611c-8f6f-4fc3-988c-93b1a25427a2/volumes" Nov 03 19:19:32.795283 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:32.795242 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:19:34.170119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:34.170080 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst"] Nov 03 19:19:34.170664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:34.170315 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" containerID="cri-o://011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8" gracePeriod=30 Nov 03 19:19:39.450481 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.450415 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 19:19:39.599753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.599719 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-data\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.599924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.599762 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istiod-ca-cert\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.599924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.599806 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-podinfo\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.599924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.599864 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-certs\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.600090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.599927 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-kube-api-access-98wlv\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.600090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.599961 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-envoy\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.600090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600058 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-socket\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.600261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600135 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-token\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-token\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.600261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600175 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-credential-socket\") pod \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\" (UID: \"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f\") " Nov 03 19:19:39.600261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600029 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-data" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.600261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600237 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-certs" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.600261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600222 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istiod-ca-cert" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:39.600558 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600338 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-socket" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.600623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600560 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-credential-socket" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.600857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600836 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.600946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600860 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.600946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600875 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.600946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600888 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.600946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.600902 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.602206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.602175 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-podinfo" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:39.602206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.602198 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-kube-api-access-98wlv" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "kube-api-access-98wlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:39.602354 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.602274 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-envoy" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:39.602526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.602504 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-token" pod "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" (UID: "fb68ae87-0d06-47ee-bd24-c2b291d0ff8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:39.666019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.665978 4948 generic.go:356] "Generic (PLEG): container finished" podID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerID="011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8" exitCode=0 Nov 03 19:19:39.666019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.666024 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" Nov 03 19:19:39.666315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.666075 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" event={"ID":"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f","Type":"ContainerDied","Data":"011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8"} Nov 03 19:19:39.666315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.666126 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst" event={"ID":"fb68ae87-0d06-47ee-bd24-c2b291d0ff8f","Type":"ContainerDied","Data":"7371717cb4990bec77cda007b36924fa41df57c08f0267a9c8d16857ea8fb3da"} Nov 03 19:19:39.666315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.666156 4948 scope.go:122] "RemoveContainer" containerID="011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8" Nov 03 19:19:39.681517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.681494 4948 scope.go:122] "RemoveContainer" containerID="011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8" Nov 03 19:19:39.681904 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:39.681882 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8\": container with ID starting with 011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8 not found: ID does not exist" containerID="011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8" Nov 03 19:19:39.681980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.681920 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8"} err="failed to get container status \"011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8\": rpc error: code = NotFound desc = could not find container \"011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8\": container with ID starting with 011c384995bf586da9814c5908bdb1955e322aae941690d501eab85c973889a8 not found: ID does not exist" Nov 03 19:19:39.702864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.702741 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98wlv\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-kube-api-access-98wlv\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.702864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.702775 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.702864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.702791 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.702864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.702801 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:39.844602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.844568 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst"] Nov 03 19:19:39.848232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:39.848206 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-cttst"] Nov 03 19:19:40.754839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:40.754798 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" path="/var/lib/kubelet/pods/fb68ae87-0d06-47ee-bd24-c2b291d0ff8f/volumes" Nov 03 19:19:43.618056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:43.618010 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w"] Nov 03 19:19:43.669801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:43.669768 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9686c3-d5t6w"] Nov 03 19:19:44.774841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:44.774799 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76af0b8-8b2e-4be5-821b-c4febd391a80" path="/var/lib/kubelet/pods/a76af0b8-8b2e-4be5-821b-c4febd391a80/volumes" Nov 03 19:19:45.197624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:45.197582 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j"] Nov 03 19:19:45.198261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:45.198190 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" containerID="cri-o://7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726" gracePeriod=30 Nov 03 19:19:49.959991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:49.959955 4948 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:19:49.960403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:49.960011 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:19:50.299446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.299408 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd"] Nov 03 19:19:50.299739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.299672 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" containerID="cri-o://a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40" gracePeriod=30 Nov 03 19:19:50.595385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.595359 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:19:50.601659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601632 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-certs\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.601832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601685 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-credential-socket\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.601832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601722 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-envoy\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.601832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601741 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-data\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601843 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istiod-ca-cert\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601869 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-podinfo\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601959 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-certs" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.601971 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-kube-api-access-vz45z\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-kube-api-access-vz45z\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602055 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-token\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602071 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-credential-socket" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602115 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-socket\") pod \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\" (UID: \"23e0c00c-5a75-46d0-8042-d43c2c776cd8\") " Nov 03 19:19:50.602179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602151 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-data" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:50.602535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602327 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istiod-ca-cert" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:50.602535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602446 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-socket" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:50.602841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602814 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.602956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602846 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.602956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602864 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.602956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602880 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.602956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.602894 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.604250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.604225 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-envoy" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:50.604332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.604240 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-kube-api-access-vz45z" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "kube-api-access-vz45z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:50.604595 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.604579 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-token" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:50.604795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.604765 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-podinfo" pod "23e0c00c-5a75-46d0-8042-d43c2c776cd8" (UID: "23e0c00c-5a75-46d0-8042-d43c2c776cd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:50.704585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.704547 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.704585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.704576 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.704585 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.704590 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e0c00c-5a75-46d0-8042-d43c2c776cd8-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.705001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.704606 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz45z\" (UniqueName: \"kubernetes.io/projected/23e0c00c-5a75-46d0-8042-d43c2c776cd8-kube-api-access-vz45z\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:50.762455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.762417 4948 generic.go:356] "Generic (PLEG): container finished" podID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerID="7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726" exitCode=0 Nov 03 19:19:50.762639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.762462 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" event={"ID":"23e0c00c-5a75-46d0-8042-d43c2c776cd8","Type":"ContainerDied","Data":"7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726"} Nov 03 19:19:50.762639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.762470 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" Nov 03 19:19:50.762639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.762509 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j" event={"ID":"23e0c00c-5a75-46d0-8042-d43c2c776cd8","Type":"ContainerDied","Data":"848aaf2bc170f9e44b370ecdfef3a1442df01dde3f838376fbbba68526d4fea8"} Nov 03 19:19:50.762639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.762539 4948 scope.go:122] "RemoveContainer" containerID="7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726" Nov 03 19:19:50.776389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.776359 4948 scope.go:122] "RemoveContainer" containerID="cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37" Nov 03 19:19:50.786531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.786512 4948 scope.go:122] "RemoveContainer" containerID="7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726" Nov 03 19:19:50.786865 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:50.786831 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726\": container with ID starting with 7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726 not found: ID does not exist" containerID="7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726" Nov 03 19:19:50.786933 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.786872 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726"} err="failed to get container status \"7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726\": rpc error: code = NotFound desc = could not find container \"7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726\": container with ID starting with 7392370fe88890cbc9ebbf483edf07f939b7c5bc346c46f01aacedc9b2ab6726 not found: ID does not exist" Nov 03 19:19:50.786933 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.786889 4948 scope.go:122] "RemoveContainer" containerID="cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37" Nov 03 19:19:50.787139 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:50.787117 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37\": container with ID starting with cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37 not found: ID does not exist" containerID="cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37" Nov 03 19:19:50.787178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.787148 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37"} err="failed to get container status \"cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37\": rpc error: code = NotFound desc = could not find container \"cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37\": container with ID starting with cc3d78a791a1575dcdb18cea51bf6f964466d44f3f445c005a54c19aec24ec37 not found: ID does not exist" Nov 03 19:19:50.932562 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.932486 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j"] Nov 03 19:19:50.937746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:50.937718 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9e277a-openshift-default-59d89f6f57-gz72j"] Nov 03 19:19:52.755853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:52.755816 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" path="/var/lib/kubelet/pods/23e0c00c-5a75-46d0-8042-d43c2c776cd8/volumes" Nov 03 19:19:55.580184 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.580115 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 19:19:55.663252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663220 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-kube-api-access-m99pz\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-kube-api-access-m99pz\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663260 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-credential-socket\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663289 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-data\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663358 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-token\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-token\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663450 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-podinfo\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663493 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istiod-ca-cert\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663512 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-envoy\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663579 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-credential-socket" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:55.663731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663591 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-certs\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.663731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663690 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-data" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:55.663731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663688 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-socket\") pod \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\" (UID: \"dd255599-9154-4ee4-b1bf-3b44fb198f9a\") " Nov 03 19:19:55.664099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.663812 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-certs" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:55.664161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.664100 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-socket" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:55.664161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.664036 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istiod-ca-cert" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:55.664380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.664359 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.664464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.664384 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.664464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.664404 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.664464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.664420 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.664464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.664434 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.665880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.665852 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-token" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:55.665980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.665953 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-podinfo" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:55.665980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.665964 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-kube-api-access-m99pz" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "kube-api-access-m99pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:55.666110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.666094 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-envoy" pod "dd255599-9154-4ee4-b1bf-3b44fb198f9a" (UID: "dd255599-9154-4ee4-b1bf-3b44fb198f9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:55.766451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.766411 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.766451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.766450 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.766739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.766466 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m99pz\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-kube-api-access-m99pz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.766739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.766482 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd255599-9154-4ee4-b1bf-3b44fb198f9a-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:55.806843 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.806802 4948 generic.go:356] "Generic (PLEG): container finished" podID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerID="a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40" exitCode=0 Nov 03 19:19:55.807000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.806861 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" Nov 03 19:19:55.807000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.806882 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" event={"ID":"dd255599-9154-4ee4-b1bf-3b44fb198f9a","Type":"ContainerDied","Data":"a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40"} Nov 03 19:19:55.807000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.806945 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd" event={"ID":"dd255599-9154-4ee4-b1bf-3b44fb198f9a","Type":"ContainerDied","Data":"5da8e0fdc7d899c9dc7b3097ea7a16c1637e07a51366e50ab2c1fc061b356909"} Nov 03 19:19:55.807000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.806968 4948 scope.go:122] "RemoveContainer" containerID="a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40" Nov 03 19:19:55.818278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.818258 4948 scope.go:122] "RemoveContainer" containerID="2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691" Nov 03 19:19:55.828629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.828608 4948 scope.go:122] "RemoveContainer" containerID="a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40" Nov 03 19:19:55.828975 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:55.828944 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40\": container with ID starting with a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40 not found: ID does not exist" containerID="a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40" Nov 03 19:19:55.829064 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.828989 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40"} err="failed to get container status \"a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40\": rpc error: code = NotFound desc = could not find container \"a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40\": container with ID starting with a2bd684ec2fe6cf9e729343806f35f89fc92c8d0d5c83eb832410617e510cc40 not found: ID does not exist" Nov 03 19:19:55.829064 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.829013 4948 scope.go:122] "RemoveContainer" containerID="2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691" Nov 03 19:19:55.829292 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:19:55.829273 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691\": container with ID starting with 2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691 not found: ID does not exist" containerID="2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691" Nov 03 19:19:55.829356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.829302 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691"} err="failed to get container status \"2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691\": rpc error: code = NotFound desc = could not find container \"2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691\": container with ID starting with 2ef6d9ac85e4c13040a1121b05c0520019cd955dba3542eff0c65624a60d1691 not found: ID does not exist" Nov 03 19:19:55.979833 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.979792 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd"] Nov 03 19:19:55.985564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:55.985535 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-wjpvd"] Nov 03 19:19:56.757424 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:19:56.757383 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" path="/var/lib/kubelet/pods/dd255599-9154-4ee4-b1bf-3b44fb198f9a/volumes" Nov 03 19:20:02.794927 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:20:02.794886 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:20:21.052670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:21.052630 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5"] Nov 03 19:20:21.102259 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:21.102220 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0568ad91-rpdn5"] Nov 03 19:20:22.754440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:22.754405 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f061e8-d50d-4099-b988-3cc62387009d" path="/var/lib/kubelet/pods/c9f061e8-d50d-4099-b988-3cc62387009d/volumes" Nov 03 19:20:32.796523 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:20:32.796429 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:20:39.716530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:39.716492 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s"] Nov 03 19:20:39.717041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:39.716766 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" containerID="cri-o://679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e" gracePeriod=30 Nov 03 19:20:43.058126 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:43.058083 4948 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:20:43.058537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:43.058165 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:20:45.017250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.017221 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:20:45.053320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053277 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-data\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053344 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-token\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-token\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053382 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-certs\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053465 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-credential-socket\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053549 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-podinfo\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053585 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99a5d5a-0550-45cf-8806-1daeb44e1730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99a5d5a-0550-45cf-8806-1daeb44e1730-istiod-ca-cert\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053617 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-data" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:45.053727 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053633 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-socket\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053750 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-certs" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:45.053963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053763 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-kube-api-access-7mrqw\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.053963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053869 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-envoy\") pod \"b99a5d5a-0550-45cf-8806-1daeb44e1730\" (UID: \"b99a5d5a-0550-45cf-8806-1daeb44e1730\") " Nov 03 19:20:45.054151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.053954 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-socket" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:45.054151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.054065 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99a5d5a-0550-45cf-8806-1daeb44e1730-istiod-ca-cert" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:45.054265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.054138 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-credential-socket" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:45.054611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.054414 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.054611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.054509 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.054611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.054531 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.054611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.054553 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99a5d5a-0550-45cf-8806-1daeb44e1730-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.054611 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.054568 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.056148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.056105 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-token" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:45.056435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.056412 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-kube-api-access-7mrqw" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "kube-api-access-7mrqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:45.056510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.056423 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-envoy" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:45.056806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.056785 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-podinfo" pod "b99a5d5a-0550-45cf-8806-1daeb44e1730" (UID: "b99a5d5a-0550-45cf-8806-1daeb44e1730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:45.155957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.155921 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.155957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.155951 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.155957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.155960 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99a5d5a-0550-45cf-8806-1daeb44e1730-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.156212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.155969 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/b99a5d5a-0550-45cf-8806-1daeb44e1730-kube-api-access-7mrqw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.251642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.251540 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p"] Nov 03 19:20:45.275903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.275866 4948 generic.go:356] "Generic (PLEG): container finished" podID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerID="679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e" exitCode=0 Nov 03 19:20:45.276080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.275920 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" Nov 03 19:20:45.276080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.275943 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" event={"ID":"b99a5d5a-0550-45cf-8806-1daeb44e1730","Type":"ContainerDied","Data":"679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e"} Nov 03 19:20:45.276080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.276007 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s" event={"ID":"b99a5d5a-0550-45cf-8806-1daeb44e1730","Type":"ContainerDied","Data":"ce390483e2fef2864b6bd11b78245d77fa9289784ca8174b080e9e379f920d8f"} Nov 03 19:20:45.276080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.276039 4948 scope.go:122] "RemoveContainer" containerID="679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e" Nov 03 19:20:45.290900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.290876 4948 scope.go:122] "RemoveContainer" containerID="8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641" Nov 03 19:20:45.302089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.302067 4948 scope.go:122] "RemoveContainer" containerID="679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e" Nov 03 19:20:45.302442 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:20:45.302420 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e\": container with ID starting with 679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e not found: ID does not exist" containerID="679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e" Nov 03 19:20:45.302497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.302452 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e"} err="failed to get container status \"679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e\": rpc error: code = NotFound desc = could not find container \"679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e\": container with ID starting with 679e715e9d18d1ef77e0486e3260b55faef32d4c57a51353750675a841b39e5e not found: ID does not exist" Nov 03 19:20:45.302497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.302472 4948 scope.go:122] "RemoveContainer" containerID="8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641" Nov 03 19:20:45.302794 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:20:45.302768 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641\": container with ID starting with 8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641 not found: ID does not exist" containerID="8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641" Nov 03 19:20:45.302869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.302800 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641"} err="failed to get container status \"8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641\": rpc error: code = NotFound desc = could not find container \"8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641\": container with ID starting with 8985c454dd64a38e4ed9196af4861d5fae93fa0e37538fccfc0179d53bbef641 not found: ID does not exist" Nov 03 19:20:45.487889 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.487848 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s"] Nov 03 19:20:45.491189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:45.491161 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ac09-openshift-default-74d79cb8cc-g995s"] Nov 03 19:20:46.755726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:46.755658 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" path="/var/lib/kubelet/pods/b99a5d5a-0550-45cf-8806-1daeb44e1730/volumes" Nov 03 19:20:52.933808 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:52.933769 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5"] Nov 03 19:20:53.043661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:53.043619 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5"] Nov 03 19:20:57.934309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:57.934261 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" podUID="11dd5f2c-306b-47fc-b243-4c2880adbdb9" containerName="payload-processing" containerID="cri-o://751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4" gracePeriod=25 Nov 03 19:20:58.044793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.044738 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" podUID="0041633e-2fd3-4de7-a336-c09632af88a4" containerName="payload-pre-processing" containerID="cri-o://fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca" gracePeriod=25 Nov 03 19:20:58.215628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.215602 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 19:20:58.344066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.344036 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dd5f2c-306b-47fc-b243-4c2880adbdb9-kube-api-access-5cm7z\" (UniqueName: \"kubernetes.io/projected/11dd5f2c-306b-47fc-b243-4c2880adbdb9-kube-api-access-5cm7z\") pod \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " Nov 03 19:20:58.344257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.344101 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dd5f2c-306b-47fc-b243-4c2880adbdb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dd5f2c-306b-47fc-b243-4c2880adbdb9-plugins-config-volume\") pod \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\" (UID: \"11dd5f2c-306b-47fc-b243-4c2880adbdb9\") " Nov 03 19:20:58.344578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.344550 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dd5f2c-306b-47fc-b243-4c2880adbdb9-plugins-config-volume" pod "11dd5f2c-306b-47fc-b243-4c2880adbdb9" (UID: "11dd5f2c-306b-47fc-b243-4c2880adbdb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:58.346250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.346220 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dd5f2c-306b-47fc-b243-4c2880adbdb9-kube-api-access-5cm7z" pod "11dd5f2c-306b-47fc-b243-4c2880adbdb9" (UID: "11dd5f2c-306b-47fc-b243-4c2880adbdb9"). InnerVolumeSpecName "kube-api-access-5cm7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:58.439787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.439666 4948 generic.go:356] "Generic (PLEG): container finished" podID="11dd5f2c-306b-47fc-b243-4c2880adbdb9" containerID="751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4" exitCode=0 Nov 03 19:20:58.439787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.439733 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" Nov 03 19:20:58.439787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.439749 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" event={"ID":"11dd5f2c-306b-47fc-b243-4c2880adbdb9","Type":"ContainerDied","Data":"751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4"} Nov 03 19:20:58.440033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.439808 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5" event={"ID":"11dd5f2c-306b-47fc-b243-4c2880adbdb9","Type":"ContainerDied","Data":"1d72ed3026b25bbb2338c0cea135dd8f85759c9e30977d85352731efa9d96d9b"} Nov 03 19:20:58.440033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.439832 4948 scope.go:122] "RemoveContainer" containerID="751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4" Nov 03 19:20:58.445250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.445228 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cm7z\" (UniqueName: \"kubernetes.io/projected/11dd5f2c-306b-47fc-b243-4c2880adbdb9-kube-api-access-5cm7z\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:58.445250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.445250 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dd5f2c-306b-47fc-b243-4c2880adbdb9-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:58.450573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.450555 4948 scope.go:122] "RemoveContainer" containerID="751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4" Nov 03 19:20:58.450904 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:20:58.450884 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4\": container with ID starting with 751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4 not found: ID does not exist" containerID="751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4" Nov 03 19:20:58.450976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.450912 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4"} err="failed to get container status \"751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4\": rpc error: code = NotFound desc = could not find container \"751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4\": container with ID starting with 751bfc8bdd364fc38955057db2f02ef6994bce30f1c78813f99b6097450ce0e4 not found: ID does not exist" Nov 03 19:20:58.609206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.609172 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5"] Nov 03 19:20:58.613345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.613318 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2e052a-c6598ff57-9w5t5"] Nov 03 19:20:58.754757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:20:58.754662 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dd5f2c-306b-47fc-b243-4c2880adbdb9" path="/var/lib/kubelet/pods/11dd5f2c-306b-47fc-b243-4c2880adbdb9/volumes" Nov 03 19:21:00.274876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.274831 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" containerID="cri-o://ec20aad21efff7db79368e70ee73b30b76616a6ef021884574e705642a65af8e" gracePeriod=45 Nov 03 19:21:00.409849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.409816 4948 scope.go:122] "RemoveContainer" containerID="0c3ce16e64f44f95529f0d53014b14cf724df586877f80b5a911fec5e12049c9" Nov 03 19:21:00.459045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.458976 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" event={"ID":"4def6a3c-a7b9-425c-8a92-c46cd6073694","Type":"ContainerDied","Data":"ec20aad21efff7db79368e70ee73b30b76616a6ef021884574e705642a65af8e"} Nov 03 19:21:00.459045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.459042 4948 generic.go:356] "Generic (PLEG): container finished" podID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerID="ec20aad21efff7db79368e70ee73b30b76616a6ef021884574e705642a65af8e" exitCode=0 Nov 03 19:21:00.566112 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.566074 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:21:00.683278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.681137 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4def6a3c-a7b9-425c-8a92-c46cd6073694-tls-certs\" (UniqueName: \"kubernetes.io/secret/4def6a3c-a7b9-425c-8a92-c46cd6073694-tls-certs\") pod \"4def6a3c-a7b9-425c-8a92-c46cd6073694\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " Nov 03 19:21:00.683278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.681752 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-home\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-home\") pod \"4def6a3c-a7b9-425c-8a92-c46cd6073694\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " Nov 03 19:21:00.683278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.681836 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-tmp-dir\") pod \"4def6a3c-a7b9-425c-8a92-c46cd6073694\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " Nov 03 19:21:00.683278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.681906 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4def6a3c-a7b9-425c-8a92-c46cd6073694-kube-api-access-xndgq\" (UniqueName: \"kubernetes.io/projected/4def6a3c-a7b9-425c-8a92-c46cd6073694-kube-api-access-xndgq\") pod \"4def6a3c-a7b9-425c-8a92-c46cd6073694\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " Nov 03 19:21:00.683278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.682027 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-dshm\") pod \"4def6a3c-a7b9-425c-8a92-c46cd6073694\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " Nov 03 19:21:00.683278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.682068 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-model-cache\") pod \"4def6a3c-a7b9-425c-8a92-c46cd6073694\" (UID: \"4def6a3c-a7b9-425c-8a92-c46cd6073694\") " Nov 03 19:21:00.683278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.682876 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-model-cache" pod "4def6a3c-a7b9-425c-8a92-c46cd6073694" (UID: "4def6a3c-a7b9-425c-8a92-c46cd6073694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.684241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.684217 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-tmp-dir" pod "4def6a3c-a7b9-425c-8a92-c46cd6073694" (UID: "4def6a3c-a7b9-425c-8a92-c46cd6073694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.685170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.685150 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-home" pod "4def6a3c-a7b9-425c-8a92-c46cd6073694" (UID: "4def6a3c-a7b9-425c-8a92-c46cd6073694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.698774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.690843 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4def6a3c-a7b9-425c-8a92-c46cd6073694-tls-certs" pod "4def6a3c-a7b9-425c-8a92-c46cd6073694" (UID: "4def6a3c-a7b9-425c-8a92-c46cd6073694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:00.698774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.698472 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-dshm" pod "4def6a3c-a7b9-425c-8a92-c46cd6073694" (UID: "4def6a3c-a7b9-425c-8a92-c46cd6073694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.703778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.702050 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4def6a3c-a7b9-425c-8a92-c46cd6073694-kube-api-access-xndgq" pod "4def6a3c-a7b9-425c-8a92-c46cd6073694" (UID: "4def6a3c-a7b9-425c-8a92-c46cd6073694"). InnerVolumeSpecName "kube-api-access-xndgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:00.784440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.784405 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.784440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.784441 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.784615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.784456 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4def6a3c-a7b9-425c-8a92-c46cd6073694-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.784615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.784469 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.784615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.784482 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4def6a3c-a7b9-425c-8a92-c46cd6073694-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.784615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:00.784495 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xndgq\" (UniqueName: \"kubernetes.io/projected/4def6a3c-a7b9-425c-8a92-c46cd6073694-kube-api-access-xndgq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:01.483712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:01.483674 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" Nov 03 19:21:01.484226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:01.483728 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p" event={"ID":"4def6a3c-a7b9-425c-8a92-c46cd6073694","Type":"ContainerDied","Data":"0a989af2fcb62387e5c015460c9d9049e871fbe00545654def616a060385c939"} Nov 03 19:21:01.484226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:01.483796 4948 scope.go:122] "RemoveContainer" containerID="ec20aad21efff7db79368e70ee73b30b76616a6ef021884574e705642a65af8e" Nov 03 19:21:01.658370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:01.658336 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p"] Nov 03 19:21:01.707902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:01.707867 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/premium-w4-2e052a-kserve-599744c99-94j4p"] Nov 03 19:21:02.753851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:02.753816 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" path="/var/lib/kubelet/pods/4def6a3c-a7b9-425c-8a92-c46cd6073694/volumes" Nov 03 19:21:02.795892 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:21:02.795852 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:21:06.426952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:06.426900 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" podUID="0041633e-2fd3-4de7-a336-c09632af88a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:21:16.427273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:16.427234 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" podUID="0041633e-2fd3-4de7-a336-c09632af88a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:21:18.334224 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.334197 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 19:21:18.407171 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.407136 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0041633e-2fd3-4de7-a336-c09632af88a4-kube-api-access-hfw77\" (UniqueName: \"kubernetes.io/projected/0041633e-2fd3-4de7-a336-c09632af88a4-kube-api-access-hfw77\") pod \"0041633e-2fd3-4de7-a336-c09632af88a4\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " Nov 03 19:21:18.407335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.407206 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0041633e-2fd3-4de7-a336-c09632af88a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041633e-2fd3-4de7-a336-c09632af88a4-plugins-config-volume\") pod \"0041633e-2fd3-4de7-a336-c09632af88a4\" (UID: \"0041633e-2fd3-4de7-a336-c09632af88a4\") " Nov 03 19:21:18.407606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.407582 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0041633e-2fd3-4de7-a336-c09632af88a4-plugins-config-volume" pod "0041633e-2fd3-4de7-a336-c09632af88a4" (UID: "0041633e-2fd3-4de7-a336-c09632af88a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:18.409343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.409317 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0041633e-2fd3-4de7-a336-c09632af88a4-kube-api-access-hfw77" pod "0041633e-2fd3-4de7-a336-c09632af88a4" (UID: "0041633e-2fd3-4de7-a336-c09632af88a4"). InnerVolumeSpecName "kube-api-access-hfw77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:18.508519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.508483 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfw77\" (UniqueName: \"kubernetes.io/projected/0041633e-2fd3-4de7-a336-c09632af88a4-kube-api-access-hfw77\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:18.508519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.508513 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0041633e-2fd3-4de7-a336-c09632af88a4-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:18.630005 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.629919 4948 generic.go:356] "Generic (PLEG): container finished" podID="0041633e-2fd3-4de7-a336-c09632af88a4" containerID="fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca" exitCode=0 Nov 03 19:21:18.630005 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.629987 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" Nov 03 19:21:18.630210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.630009 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" event={"ID":"0041633e-2fd3-4de7-a336-c09632af88a4","Type":"ContainerDied","Data":"fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca"} Nov 03 19:21:18.630210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.630078 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5" event={"ID":"0041633e-2fd3-4de7-a336-c09632af88a4","Type":"ContainerDied","Data":"479b814abfacf21c661a59451b388307d2c2a2de726b9948250df86b102e8fd0"} Nov 03 19:21:18.630210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.630107 4948 scope.go:122] "RemoveContainer" containerID="fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca" Nov 03 19:21:18.642563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.642536 4948 scope.go:122] "RemoveContainer" containerID="fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca" Nov 03 19:21:18.642901 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:21:18.642877 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca\": container with ID starting with fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca not found: ID does not exist" containerID="fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca" Nov 03 19:21:18.643001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.642911 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca"} err="failed to get container status \"fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca\": rpc error: code = NotFound desc = could not find container \"fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca\": container with ID starting with fde21f301218d06920a259eec63f70ea99d152d9a5e91179fa252956052985ca not found: ID does not exist" Nov 03 19:21:18.801778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.801750 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5"] Nov 03 19:21:18.805019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:18.804996 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2e052a-564f4b8d67-mz5p5"] Nov 03 19:21:20.755012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:20.754973 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0041633e-2fd3-4de7-a336-c09632af88a4" path="/var/lib/kubelet/pods/0041633e-2fd3-4de7-a336-c09632af88a4/volumes" Nov 03 19:21:28.168015 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:28.167976 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2"] Nov 03 19:21:32.793507 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:21:32.793472 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:21:33.169025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:33.168941 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" podUID="2ce424f6-afbd-46af-af02-3a40594a498c" containerName="payload-processing" containerID="cri-o://180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63" gracePeriod=25 Nov 03 19:21:41.761891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:41.761848 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" podUID="2ce424f6-afbd-46af-af02-3a40594a498c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:21:51.761863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:51.761827 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" podUID="2ce424f6-afbd-46af-af02-3a40594a498c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:21:53.438975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.438951 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:21:53.548896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.548860 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce424f6-afbd-46af-af02-3a40594a498c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce424f6-afbd-46af-af02-3a40594a498c-plugins-config-volume\") pod \"2ce424f6-afbd-46af-af02-3a40594a498c\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " Nov 03 19:21:53.549082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.548986 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce424f6-afbd-46af-af02-3a40594a498c-kube-api-access-dr2qb\" (UniqueName: \"kubernetes.io/projected/2ce424f6-afbd-46af-af02-3a40594a498c-kube-api-access-dr2qb\") pod \"2ce424f6-afbd-46af-af02-3a40594a498c\" (UID: \"2ce424f6-afbd-46af-af02-3a40594a498c\") " Nov 03 19:21:53.549307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.549285 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce424f6-afbd-46af-af02-3a40594a498c-plugins-config-volume" pod "2ce424f6-afbd-46af-af02-3a40594a498c" (UID: "2ce424f6-afbd-46af-af02-3a40594a498c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:53.549404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.549389 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce424f6-afbd-46af-af02-3a40594a498c-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:53.551125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.551096 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce424f6-afbd-46af-af02-3a40594a498c-kube-api-access-dr2qb" pod "2ce424f6-afbd-46af-af02-3a40594a498c" (UID: "2ce424f6-afbd-46af-af02-3a40594a498c"). InnerVolumeSpecName "kube-api-access-dr2qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:53.651116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.651083 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr2qb\" (UniqueName: \"kubernetes.io/projected/2ce424f6-afbd-46af-af02-3a40594a498c-kube-api-access-dr2qb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:53.922834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.922750 4948 generic.go:356] "Generic (PLEG): container finished" podID="2ce424f6-afbd-46af-af02-3a40594a498c" containerID="180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63" exitCode=0 Nov 03 19:21:53.922834 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.922798 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" Nov 03 19:21:53.923035 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.922838 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" event={"ID":"2ce424f6-afbd-46af-af02-3a40594a498c","Type":"ContainerDied","Data":"180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63"} Nov 03 19:21:53.923035 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.922893 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2" event={"ID":"2ce424f6-afbd-46af-af02-3a40594a498c","Type":"ContainerDied","Data":"396b18f3892077849090498827dd7a8a806dc260156683f1c6201c77fdf3edc6"} Nov 03 19:21:53.923035 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.922917 4948 scope.go:122] "RemoveContainer" containerID="180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63" Nov 03 19:21:53.933836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.933817 4948 scope.go:122] "RemoveContainer" containerID="180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63" Nov 03 19:21:53.934130 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:21:53.934108 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63\": container with ID starting with 180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63 not found: ID does not exist" containerID="180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63" Nov 03 19:21:53.934196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:53.934139 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63"} err="failed to get container status \"180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63\": rpc error: code = NotFound desc = could not find container \"180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63\": container with ID starting with 180c87c26d9f6ba519d18ca21c365c438a9700f67a9c3b9608638bb7beec4e63 not found: ID does not exist" Nov 03 19:21:54.092564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:54.092522 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2"] Nov 03 19:21:54.143117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:54.143079 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-wlgw2"] Nov 03 19:21:54.755159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:21:54.755128 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce424f6-afbd-46af-af02-3a40594a498c" path="/var/lib/kubelet/pods/2ce424f6-afbd-46af-af02-3a40594a498c/volumes" Nov 03 19:22:00.454145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:00.454111 4948 scope.go:122] "RemoveContainer" containerID="c26742cf1c5dbd6dd13ab071661826ee54d2fb839fefcd81ba82fb0c521d41e0" Nov 03 19:22:00.464592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:00.464566 4948 scope.go:122] "RemoveContainer" containerID="125c7d122a2dc7722297558f452a3d5c1a0dd1555df12adda14780d086a1e238" Nov 03 19:22:00.474876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:00.474852 4948 scope.go:122] "RemoveContainer" containerID="b27588a904c574cfe3a383759f42eac5bad9b0cf41afb9def311c5c8227fecdf" Nov 03 19:22:02.652984 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.652947 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk"] Nov 03 19:22:02.653451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.653171 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" podUID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" containerName="discovery" containerID="cri-o://2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc" gracePeriod=30 Nov 03 19:22:02.767194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.767161 4948 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qjtdk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:8080/ready\": dial tcp 10.132.0.51:8080: connect: connection refused" start-of-body= Nov 03 19:22:02.767361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.767224 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" podUID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.51:8080/ready\": dial tcp 10.132.0.51:8080: connect: connection refused" Nov 03 19:22:02.804604 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:22:02.804476 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:22:02.932193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.932167 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 19:22:02.964980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.964950 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-token\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-token\") pod \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " Nov 03 19:22:02.965157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.964998 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-ca-configmap\") pod \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " Nov 03 19:22:02.965157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.965027 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-kube-api-access-jw7gp\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-kube-api-access-jw7gp\") pod \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " Nov 03 19:22:02.965254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.965209 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-kubeconfig\") pod \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " Nov 03 19:22:02.965305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.965265 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e929ed-5855-4d0e-98f3-c9c0351c53ab-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e929ed-5855-4d0e-98f3-c9c0351c53ab-local-certs\") pod \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " Nov 03 19:22:02.965356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.965312 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-cacerts\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-cacerts\") pod \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " Nov 03 19:22:02.965523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.965404 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-ca-configmap" pod "47e929ed-5855-4d0e-98f3-c9c0351c53ab" (UID: "47e929ed-5855-4d0e-98f3-c9c0351c53ab"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:02.965523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.965422 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-dns-cert\") pod \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\" (UID: \"47e929ed-5855-4d0e-98f3-c9c0351c53ab\") " Nov 03 19:22:02.966004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.965963 4948 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-ca-configmap\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:02.967988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.967962 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-kube-api-access-jw7gp" pod "47e929ed-5855-4d0e-98f3-c9c0351c53ab" (UID: "47e929ed-5855-4d0e-98f3-c9c0351c53ab"). InnerVolumeSpecName "kube-api-access-jw7gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:02.968118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.968094 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-kubeconfig" pod "47e929ed-5855-4d0e-98f3-c9c0351c53ab" (UID: "47e929ed-5855-4d0e-98f3-c9c0351c53ab"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:02.968347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.968321 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-token" pod "47e929ed-5855-4d0e-98f3-c9c0351c53ab" (UID: "47e929ed-5855-4d0e-98f3-c9c0351c53ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:02.968544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.968519 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e929ed-5855-4d0e-98f3-c9c0351c53ab-local-certs" pod "47e929ed-5855-4d0e-98f3-c9c0351c53ab" (UID: "47e929ed-5855-4d0e-98f3-c9c0351c53ab"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:02.968544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.968533 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-cacerts" pod "47e929ed-5855-4d0e-98f3-c9c0351c53ab" (UID: "47e929ed-5855-4d0e-98f3-c9c0351c53ab"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:02.968798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:02.968775 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-dns-cert" pod "47e929ed-5855-4d0e-98f3-c9c0351c53ab" (UID: "47e929ed-5855-4d0e-98f3-c9c0351c53ab"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:03.008444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.007808 4948 generic.go:356] "Generic (PLEG): container finished" podID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" containerID="2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc" exitCode=0 Nov 03 19:22:03.008444 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.007904 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" Nov 03 19:22:03.008756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.008445 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" event={"ID":"47e929ed-5855-4d0e-98f3-c9c0351c53ab","Type":"ContainerDied","Data":"2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc"} Nov 03 19:22:03.008756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.008508 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk" event={"ID":"47e929ed-5855-4d0e-98f3-c9c0351c53ab","Type":"ContainerDied","Data":"43db91dd4f2714a1d4447bedfe202eae300c7746dcde13e3b1d867a6f2d2a555"} Nov 03 19:22:03.008756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.008543 4948 scope.go:122] "RemoveContainer" containerID="2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc" Nov 03 19:22:03.031196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.031168 4948 scope.go:122] "RemoveContainer" containerID="2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc" Nov 03 19:22:03.031687 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:22:03.031650 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc\": container with ID starting with 2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc not found: ID does not exist" containerID="2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc" Nov 03 19:22:03.031838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.031695 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc"} err="failed to get container status \"2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc\": rpc error: code = NotFound desc = could not find container \"2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc\": container with ID starting with 2dfdf860b5fdf2cb2cbae65c7d23f5c66331470e99f902173c0ffb0d341364fc not found: ID does not exist" Nov 03 19:22:03.068485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.067911 4948 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-kubeconfig\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.068485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.067948 4948 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e929ed-5855-4d0e-98f3-c9c0351c53ab-local-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.068485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.067963 4948 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-cacerts\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.068485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.067986 4948 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-csr-dns-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.068485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.067998 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.068485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.068012 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7gp\" (UniqueName: \"kubernetes.io/projected/47e929ed-5855-4d0e-98f3-c9c0351c53ab-kube-api-access-jw7gp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.186294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.186265 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk"] Nov 03 19:22:03.191313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:03.191288 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjtdk"] Nov 03 19:22:04.754485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:04.754450 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" path="/var/lib/kubelet/pods/47e929ed-5855-4d0e-98f3-c9c0351c53ab/volumes" Nov 03 19:22:32.794135 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:22:32.794098 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:22:38.075023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.074988 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96"] Nov 03 19:22:38.075535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.075192 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" podUID="e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" containerName="maas-api" containerID="cri-o://211e43d8387d0ab870142f0bbd236952c47b58e0695512f137f9603cba99202b" gracePeriod=30 Nov 03 19:22:38.317457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.317419 4948 generic.go:356] "Generic (PLEG): container finished" podID="e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" containerID="211e43d8387d0ab870142f0bbd236952c47b58e0695512f137f9603cba99202b" exitCode=0 Nov 03 19:22:38.317663 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.317510 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" event={"ID":"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f","Type":"ContainerDied","Data":"211e43d8387d0ab870142f0bbd236952c47b58e0695512f137f9603cba99202b"} Nov 03 19:22:38.365183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.365158 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:22:38.508389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.508352 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-openshift-service-ca\") pod \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " Nov 03 19:22:38.508582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.508566 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-kube-api-access-9cmgk\" (UniqueName: \"kubernetes.io/projected/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-kube-api-access-9cmgk\") pod \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " Nov 03 19:22:38.508637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.508596 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls\") pod \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " Nov 03 19:22:38.508637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.508617 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-metrics-certs\") pod \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\" (UID: \"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f\") " Nov 03 19:22:38.509437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.509403 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-openshift-service-ca" pod "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" (UID: "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:38.511008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.510969 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-metrics-certs" pod "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" (UID: "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:38.511206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.511180 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-kube-api-access-9cmgk" pod "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" (UID: "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f"). InnerVolumeSpecName "kube-api-access-9cmgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:38.511298 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.511223 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls" pod "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" (UID: "e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:38.610212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.610106 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:38.610212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.610154 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:38.610212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.610172 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmgk\" (UniqueName: \"kubernetes.io/projected/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-kube-api-access-9cmgk\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:38.610212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:38.610187 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:39.325784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:39.325751 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" event={"ID":"e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f","Type":"ContainerDied","Data":"6054cbdbf5b240a56472d2e629e75d3a1ec9275ee669dede2842f247667cf225"} Nov 03 19:22:39.326277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:39.325796 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96" Nov 03 19:22:39.326277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:39.325823 4948 scope.go:122] "RemoveContainer" containerID="211e43d8387d0ab870142f0bbd236952c47b58e0695512f137f9603cba99202b" Nov 03 19:22:39.495858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:39.495794 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96"] Nov 03 19:22:39.546640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:39.546602 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-h4m96"] Nov 03 19:22:40.754300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:40.754267 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" path="/var/lib/kubelet/pods/e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f/volumes" Nov 03 19:22:50.588905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.588180 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc"] Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589062 4948 cpu_manager.go:414] "containerMap: removing container" podUID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" containerName="payload-pre-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589086 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" containerName="payload-pre-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589106 4948 cpu_manager.go:414] "containerMap: removing container" podUID="e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" containerName="maas-api" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589115 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" containerName="maas-api" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589134 4948 cpu_manager.go:414] "containerMap: removing container" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589143 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589155 4948 cpu_manager.go:414] "containerMap: removing container" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589162 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589177 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589185 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589196 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589203 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589223 4948 cpu_manager.go:414] "containerMap: removing container" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589229 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589240 4948 cpu_manager.go:414] "containerMap: removing container" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589247 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589260 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2ce424f6-afbd-46af-af02-3a40594a498c" containerName="payload-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589267 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce424f6-afbd-46af-af02-3a40594a498c" containerName="payload-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589286 4948 cpu_manager.go:414] "containerMap: removing container" podUID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" containerName="discovery" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589293 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" containerName="discovery" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589308 4948 cpu_manager.go:414] "containerMap: removing container" podUID="11dd5f2c-306b-47fc-b243-4c2880adbdb9" containerName="payload-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589314 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dd5f2c-306b-47fc-b243-4c2880adbdb9" containerName="payload-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589331 4948 cpu_manager.go:414] "containerMap: removing container" podUID="0041633e-2fd3-4de7-a336-c09632af88a4" containerName="payload-pre-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589339 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0041633e-2fd3-4de7-a336-c09632af88a4" containerName="payload-pre-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589350 4948 cpu_manager.go:414] "containerMap: removing container" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589357 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589369 4948 cpu_manager.go:414] "containerMap: removing container" podUID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" containerName="payload-pre-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589376 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" containerName="payload-pre-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589397 4948 cpu_manager.go:414] "containerMap: removing container" podUID="98a3e420-71c3-4cca-9036-323dfd7f1eba" containerName="payload-processing" Nov 03 19:22:50.589412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589404 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a3e420-71c3-4cca-9036-323dfd7f1eba" containerName="payload-processing" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589524 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a3e420-71c3-4cca-9036-323dfd7f1eba" containerName="payload-processing" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589537 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc2b876-e5c6-4008-9558-f3568dc1a6e8" containerName="payload-pre-processing" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589550 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b6f2bf-8dfb-4c2a-a1d8-91c044c4bc7f" containerName="maas-api" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589559 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589568 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589577 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589586 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4def6a3c-a7b9-425c-8a92-c46cd6073694" containerName="main" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589596 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd255599-9154-4ee4-b1bf-3b44fb198f9a" containerName="istio-proxy" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589604 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e0c00c-5a75-46d0-8042-d43c2c776cd8" containerName="istio-proxy" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589615 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce424f6-afbd-46af-af02-3a40594a498c" containerName="payload-processing" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589625 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e929ed-5855-4d0e-98f3-c9c0351c53ab" containerName="discovery" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589635 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dd5f2c-306b-47fc-b243-4c2880adbdb9" containerName="payload-processing" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589645 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589655 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="0041633e-2fd3-4de7-a336-c09632af88a4" containerName="payload-pre-processing" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589664 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb68ae87-0d06-47ee-bd24-c2b291d0ff8f" containerName="istio-proxy" Nov 03 19:22:50.590471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.589674 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd2d734-ddb8-47d5-8a3d-c149f684b6d4" containerName="payload-pre-processing" Nov 03 19:22:50.602507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.602471 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:50.612661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.612629 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc"] Nov 03 19:22:50.707732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.707673 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3d74f801-dockercfg-8dsvb\"" Nov 03 19:22:50.707732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.707688 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3d74f801\"" Nov 03 19:22:50.739781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.739741 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx"] Nov 03 19:22:50.740365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.740346 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" Nov 03 19:22:50.740466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.740367 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99a5d5a-0550-45cf-8806-1daeb44e1730" containerName="istio-proxy" Nov 03 19:22:50.743940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.743918 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:50.758396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.758367 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28d4dd5-c3df-4b8f-918c-527d662015da-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:50.758504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.758401 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrkn\" (UniqueName: \"kubernetes.io/projected/c28d4dd5-c3df-4b8f-918c-527d662015da-kube-api-access-dqrkn\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:50.800440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.800402 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx"] Nov 03 19:22:50.859997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.859907 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6kq\" (UniqueName: \"kubernetes.io/projected/d7085f31-a663-487c-8683-a3ddc958e1de-kube-api-access-tc6kq\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:50.860229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.860042 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28d4dd5-c3df-4b8f-918c-527d662015da-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:50.860229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.860080 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrkn\" (UniqueName: \"kubernetes.io/projected/c28d4dd5-c3df-4b8f-918c-527d662015da-kube-api-access-dqrkn\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:50.860229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.860120 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7085f31-a663-487c-8683-a3ddc958e1de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:50.912648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.912618 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3d74f801\"" Nov 03 19:22:50.920874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.920845 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28d4dd5-c3df-4b8f-918c-527d662015da-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:50.923017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.922988 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrkn\" (UniqueName: \"kubernetes.io/projected/c28d4dd5-c3df-4b8f-918c-527d662015da-kube-api-access-dqrkn\") pod \"payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:50.961492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.961456 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6kq\" (UniqueName: \"kubernetes.io/projected/d7085f31-a663-487c-8683-a3ddc958e1de-kube-api-access-tc6kq\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:50.961686 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.961555 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7085f31-a663-487c-8683-a3ddc958e1de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:50.962306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.962285 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7085f31-a663-487c-8683-a3ddc958e1de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:50.979803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.979779 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3d74f801-dockercfg-8dsvb\"" Nov 03 19:22:50.987960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:50.987938 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:51.024733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:51.024678 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6kq\" (UniqueName: \"kubernetes.io/projected/d7085f31-a663-487c-8683-a3ddc958e1de-kube-api-access-tc6kq\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:51.060893 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:51.060856 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:51.379792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:51.379723 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc"] Nov 03 19:22:51.380255 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:22:51.380225 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28d4dd5_c3df_4b8f_918c_527d662015da.slice/crio-9ad7275e83c9c39a01f94afcce3913d16083a5479254996b8f72432be3c811a8 WatchSource:0}: Error finding container 9ad7275e83c9c39a01f94afcce3913d16083a5479254996b8f72432be3c811a8: Status 404 returned error can't find the container with id 9ad7275e83c9c39a01f94afcce3913d16083a5479254996b8f72432be3c811a8 Nov 03 19:22:51.417063 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:51.417036 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" event={"ID":"c28d4dd5-c3df-4b8f-918c-527d662015da","Type":"ContainerStarted","Data":"9ad7275e83c9c39a01f94afcce3913d16083a5479254996b8f72432be3c811a8"} Nov 03 19:22:51.482457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:51.482427 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx"] Nov 03 19:22:51.482954 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:22:51.482927 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7085f31_a663_487c_8683_a3ddc958e1de.slice/crio-ed8f3c2891b62fe97066d83e387efd7cf236fbfd309329f1d0dc940dec1d2749 WatchSource:0}: Error finding container ed8f3c2891b62fe97066d83e387efd7cf236fbfd309329f1d0dc940dec1d2749: Status 404 returned error can't find the container with id ed8f3c2891b62fe97066d83e387efd7cf236fbfd309329f1d0dc940dec1d2749 Nov 03 19:22:52.428048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:52.428004 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" event={"ID":"c28d4dd5-c3df-4b8f-918c-527d662015da","Type":"ContainerStarted","Data":"5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265"} Nov 03 19:22:52.428531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:52.428290 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:22:52.437997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:52.437957 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" event={"ID":"d7085f31-a663-487c-8683-a3ddc958e1de","Type":"ContainerStarted","Data":"ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb"} Nov 03 19:22:52.438172 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:52.438039 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" event={"ID":"d7085f31-a663-487c-8683-a3ddc958e1de","Type":"ContainerStarted","Data":"ed8f3c2891b62fe97066d83e387efd7cf236fbfd309329f1d0dc940dec1d2749"} Nov 03 19:22:52.438290 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:52.438269 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:22:52.548487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:52.548436 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" podStartSLOduration=2.548421813 podStartE2EDuration="2.548421813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:22:52.546523012 +0000 UTC m=+2702.029518926" watchObservedRunningTime="2026-11-03 19:22:52.548421813 +0000 UTC m=+2702.031417727" Nov 03 19:22:52.665124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:52.665070 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" podStartSLOduration=2.665057548 podStartE2EDuration="2.665057548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:22:52.664154545 +0000 UTC m=+2702.147150459" watchObservedRunningTime="2026-11-03 19:22:52.665057548 +0000 UTC m=+2702.148053461" Nov 03 19:22:55.635693 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:55.635665 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:22:55.643495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:55.643474 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:22:55.899441 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:55.899351 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:22:55.905825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:22:55.905801 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:23:00.554004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:00.553972 4948 scope.go:122] "RemoveContainer" containerID="e3bef5b59256c8a00b20106a605dee73f5f9d311701beaec5759e9c5ac580675" Nov 03 19:23:02.794622 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:23:02.794575 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:23:03.448629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:03.448600 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:23:03.448827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:03.448679 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:23:12.227809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.227770 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv"] Nov 03 19:23:12.233651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.233629 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.287232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.287199 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv"] Nov 03 19:23:12.308557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.308521 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lvz\" (UniqueName: \"kubernetes.io/projected/7298fafe-3b44-4a92-83a6-461bf774367d-kube-api-access-r6lvz\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.308744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.308608 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.308744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.308676 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7298fafe-3b44-4a92-83a6-461bf774367d-openshift-service-ca\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.308861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.308787 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-metrics-certs\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.333878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.333855 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d74f801-serving-cert\"" Nov 03 19:23:12.410226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.410199 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lvz\" (UniqueName: \"kubernetes.io/projected/7298fafe-3b44-4a92-83a6-461bf774367d-kube-api-access-r6lvz\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.410430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.410241 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.410430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.410311 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7298fafe-3b44-4a92-83a6-461bf774367d-openshift-service-ca\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.410430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.410385 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-metrics-certs\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.410430 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:23:12.410420 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-serving-cert: secret "maas-api-e2e-mt-3d74f801-serving-cert" not found Nov 03 19:23:12.410678 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:23:12.410495 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls podName:7298fafe-3b44-4a92-83a6-461bf774367d nodeName:}" failed. No retries permitted until 2026-11-03 19:23:12.910475521 +0000 UTC m=+2722.393471417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls") pod "maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" (UID: "7298fafe-3b44-4a92-83a6-461bf774367d") : secret "maas-api-e2e-mt-3d74f801-serving-cert" not found Nov 03 19:23:12.410993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.410969 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7298fafe-3b44-4a92-83a6-461bf774367d-openshift-service-ca\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.412906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.412879 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-metrics-certs\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.474962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.474920 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lvz\" (UniqueName: \"kubernetes.io/projected/7298fafe-3b44-4a92-83a6-461bf774367d-kube-api-access-r6lvz\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.915924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.915888 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:12.918341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:12.918309 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls\") pod \"maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:13.148439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:13.148396 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:13.536802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:13.536767 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv"] Nov 03 19:23:13.537424 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:23:13.537392 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7298fafe_3b44_4a92_83a6_461bf774367d.slice/crio-794c27f6f8ad5d53e0217d1f5f163e53c4f43a899e7f2aa33571ae20903ea165 WatchSource:0}: Error finding container 794c27f6f8ad5d53e0217d1f5f163e53c4f43a899e7f2aa33571ae20903ea165: Status 404 returned error can't find the container with id 794c27f6f8ad5d53e0217d1f5f163e53c4f43a899e7f2aa33571ae20903ea165 Nov 03 19:23:13.538962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:13.538937 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:23:13.598853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:13.598817 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" event={"ID":"7298fafe-3b44-4a92-83a6-461bf774367d","Type":"ContainerStarted","Data":"794c27f6f8ad5d53e0217d1f5f163e53c4f43a899e7f2aa33571ae20903ea165"} Nov 03 19:23:14.607405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:14.607362 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" event={"ID":"7298fafe-3b44-4a92-83a6-461bf774367d","Type":"ContainerStarted","Data":"0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00"} Nov 03 19:23:14.607842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:14.607603 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:14.727844 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:14.727780 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" podStartSLOduration=1.902402075 podStartE2EDuration="2.727766181s" totalImagesPullingTime="825.364106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:14.726360423 +0000 UTC m=+2724.209356351" watchObservedRunningTime="2026-11-03 19:23:14.727766181 +0000 UTC m=+2724.210762089" Nov 03 19:23:15.052666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:15.052574 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf"] Nov 03 19:23:15.059633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:15.059594 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da52e3b-xt2rf"] Nov 03 19:23:16.754649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:16.754603 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84c93f8-40ee-4fdc-8530-2ed0885f70da" path="/var/lib/kubelet/pods/f84c93f8-40ee-4fdc-8530-2ed0885f70da/volumes" Nov 03 19:23:20.620473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:20.620440 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:23:28.054692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:28.054660 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js"] Nov 03 19:23:28.105594 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:28.105562 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c07c1f-j29js"] Nov 03 19:23:28.754494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:28.754461 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1060ced-d832-4f84-9e91-f52de3558b48" path="/var/lib/kubelet/pods/a1060ced-d832-4f84-9e91-f52de3558b48/volumes" Nov 03 19:23:32.792761 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:23:32.792657 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:23:53.051595 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:53.051560 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2"] Nov 03 19:23:53.118591 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:53.118538 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d4b55a-5t5b2"] Nov 03 19:23:54.753963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:23:54.753924 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d11faf-4101-49c8-a416-d60bd5147fe3" path="/var/lib/kubelet/pods/b7d11faf-4101-49c8-a416-d60bd5147fe3/volumes" Nov 03 19:24:00.614320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:00.614286 4948 scope.go:122] "RemoveContainer" containerID="49f49af8127358b3fe13fef1c162f7da56a1d05236975a577782740cf23b608e" Nov 03 19:24:01.530462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.530430 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-643cff"] Nov 03 19:24:01.535795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.535766 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:01.585103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.585068 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-643cff"] Nov 03 19:24:01.602404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.602368 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7v5\" (UniqueName: \"kubernetes.io/projected/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6-kube-api-access-gd7v5\") pod \"test-curl-7315-643cff\" (UID: \"8752d7c3-c9a1-42c9-b803-ace2eaa2aef6\") " pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:01.632142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.632109 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zmjtl\"" Nov 03 19:24:01.703811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.703768 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7v5\" (UniqueName: \"kubernetes.io/projected/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6-kube-api-access-gd7v5\") pod \"test-curl-7315-643cff\" (UID: \"8752d7c3-c9a1-42c9-b803-ace2eaa2aef6\") " pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:01.766175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.766143 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7v5\" (UniqueName: \"kubernetes.io/projected/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6-kube-api-access-gd7v5\") pod \"test-curl-7315-643cff\" (UID: \"8752d7c3-c9a1-42c9-b803-ace2eaa2aef6\") " pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:01.848789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:01.848711 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:02.224740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:02.224694 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-643cff"] Nov 03 19:24:02.225624 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:24:02.225596 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8752d7c3_c9a1_42c9_b803_ace2eaa2aef6.slice/crio-22b20bec217ab3111e9f78fcd9dd6d486139484a125e15cc864707f767781b63 WatchSource:0}: Error finding container 22b20bec217ab3111e9f78fcd9dd6d486139484a125e15cc864707f767781b63: Status 404 returned error can't find the container with id 22b20bec217ab3111e9f78fcd9dd6d486139484a125e15cc864707f767781b63 Nov 03 19:24:02.794801 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:02.794763 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:24:02.972207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:02.972168 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-643cff" event={"ID":"8752d7c3-c9a1-42c9-b803-ace2eaa2aef6","Type":"ContainerStarted","Data":"22b20bec217ab3111e9f78fcd9dd6d486139484a125e15cc864707f767781b63"} Nov 03 19:24:06.008487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:06.008383 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-643cff" event={"ID":"8752d7c3-c9a1-42c9-b803-ace2eaa2aef6","Type":"ContainerStarted","Data":"884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35"} Nov 03 19:24:06.128023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:06.127968 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-643cff" podStartSLOduration=1.6149310510000001 podStartE2EDuration="5.127953284s" totalImagesPullingTime="3.513022233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:06.124387165 +0000 UTC m=+2775.607383080" watchObservedRunningTime="2026-11-03 19:24:06.127953284 +0000 UTC m=+2775.610949197" Nov 03 19:24:07.665833 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:07.665795 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-643cff"] Nov 03 19:24:07.670828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:07.670791 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-643cff"] Nov 03 19:24:08.022060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:08.022018 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-643cff" podUID="8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" containerName="test-curl-7315-643cff" containerID="cri-o://884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35" gracePeriod=2 Nov 03 19:24:08.074322 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:08.074283 4948 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-643cff\" is forbidden: User \"system:node:ip-10-0-128-95.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-95.us-west-2.compute.internal' and this object" podUID="8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:10.282074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:10.282009 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:10.404047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:10.404009 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6-kube-api-access-gd7v5\" (UniqueName: \"kubernetes.io/projected/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6-kube-api-access-gd7v5\") pod \"8752d7c3-c9a1-42c9-b803-ace2eaa2aef6\" (UID: \"8752d7c3-c9a1-42c9-b803-ace2eaa2aef6\") " Nov 03 19:24:10.406140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:10.406103 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6-kube-api-access-gd7v5" pod "8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" (UID: "8752d7c3-c9a1-42c9-b803-ace2eaa2aef6"). InnerVolumeSpecName "kube-api-access-gd7v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:10.505910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:10.505877 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd7v5\" (UniqueName: \"kubernetes.io/projected/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6-kube-api-access-gd7v5\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:10.754331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:10.754293 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" path="/var/lib/kubelet/pods/8752d7c3-c9a1-42c9-b803-ace2eaa2aef6/volumes" Nov 03 19:24:11.047139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:11.047053 4948 generic.go:356] "Generic (PLEG): container finished" podID="8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" containerID="884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35" exitCode=137 Nov 03 19:24:11.047139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:11.047103 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-643cff" Nov 03 19:24:11.047356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:11.047149 4948 scope.go:122] "RemoveContainer" containerID="884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35" Nov 03 19:24:11.059638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:11.059617 4948 scope.go:122] "RemoveContainer" containerID="884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35" Nov 03 19:24:11.059954 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:11.059927 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35\": container with ID starting with 884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35 not found: ID does not exist" containerID="884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35" Nov 03 19:24:11.060050 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:11.059964 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35"} err="failed to get container status \"884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35\": rpc error: code = NotFound desc = could not find container \"884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35\": container with ID starting with 884d289a331ad7694b3ce1abf929fd1db915d836271b13480e72b3ee887f8c35 not found: ID does not exist" Nov 03 19:24:20.557267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:20.557229 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc"] Nov 03 19:24:20.668937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:20.668895 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx"] Nov 03 19:24:25.558093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.558042 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" podUID="c28d4dd5-c3df-4b8f-918c-527d662015da" containerName="payload-processing" containerID="cri-o://5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265" gracePeriod=25 Nov 03 19:24:25.669566 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.669515 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" podUID="d7085f31-a663-487c-8683-a3ddc958e1de" containerName="payload-pre-processing" containerID="cri-o://ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb" gracePeriod=25 Nov 03 19:24:25.838929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.838894 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:24:25.910103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.910075 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28d4dd5-c3df-4b8f-918c-527d662015da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28d4dd5-c3df-4b8f-918c-527d662015da-plugins-config-volume\") pod \"c28d4dd5-c3df-4b8f-918c-527d662015da\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " Nov 03 19:24:25.910256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.910242 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28d4dd5-c3df-4b8f-918c-527d662015da-kube-api-access-dqrkn\" (UniqueName: \"kubernetes.io/projected/c28d4dd5-c3df-4b8f-918c-527d662015da-kube-api-access-dqrkn\") pod \"c28d4dd5-c3df-4b8f-918c-527d662015da\" (UID: \"c28d4dd5-c3df-4b8f-918c-527d662015da\") " Nov 03 19:24:25.910433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.910405 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28d4dd5-c3df-4b8f-918c-527d662015da-plugins-config-volume" pod "c28d4dd5-c3df-4b8f-918c-527d662015da" (UID: "c28d4dd5-c3df-4b8f-918c-527d662015da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:25.910649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.910625 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28d4dd5-c3df-4b8f-918c-527d662015da-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:25.912386 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.912363 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28d4dd5-c3df-4b8f-918c-527d662015da-kube-api-access-dqrkn" pod "c28d4dd5-c3df-4b8f-918c-527d662015da" (UID: "c28d4dd5-c3df-4b8f-918c-527d662015da"). InnerVolumeSpecName "kube-api-access-dqrkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:25.920234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:25.920207 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:24:26.011387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.011362 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7085f31-a663-487c-8683-a3ddc958e1de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7085f31-a663-487c-8683-a3ddc958e1de-plugins-config-volume\") pod \"d7085f31-a663-487c-8683-a3ddc958e1de\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " Nov 03 19:24:26.011547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.011406 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7085f31-a663-487c-8683-a3ddc958e1de-kube-api-access-tc6kq\" (UniqueName: \"kubernetes.io/projected/d7085f31-a663-487c-8683-a3ddc958e1de-kube-api-access-tc6kq\") pod \"d7085f31-a663-487c-8683-a3ddc958e1de\" (UID: \"d7085f31-a663-487c-8683-a3ddc958e1de\") " Nov 03 19:24:26.011670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.011656 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrkn\" (UniqueName: \"kubernetes.io/projected/c28d4dd5-c3df-4b8f-918c-527d662015da-kube-api-access-dqrkn\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:26.011894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.011867 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7085f31-a663-487c-8683-a3ddc958e1de-plugins-config-volume" pod "d7085f31-a663-487c-8683-a3ddc958e1de" (UID: "d7085f31-a663-487c-8683-a3ddc958e1de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:26.013586 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.013561 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7085f31-a663-487c-8683-a3ddc958e1de-kube-api-access-tc6kq" pod "d7085f31-a663-487c-8683-a3ddc958e1de" (UID: "d7085f31-a663-487c-8683-a3ddc958e1de"). InnerVolumeSpecName "kube-api-access-tc6kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:26.055084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.055050 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv"] Nov 03 19:24:26.055258 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.055239 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" podUID="7298fafe-3b44-4a92-83a6-461bf774367d" containerName="maas-api" containerID="cri-o://0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00" gracePeriod=30 Nov 03 19:24:26.112356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.112327 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7085f31-a663-487c-8683-a3ddc958e1de-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:26.112356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.112354 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc6kq\" (UniqueName: \"kubernetes.io/projected/d7085f31-a663-487c-8683-a3ddc958e1de-kube-api-access-tc6kq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:26.166266 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.166233 4948 generic.go:356] "Generic (PLEG): container finished" podID="d7085f31-a663-487c-8683-a3ddc958e1de" containerID="ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb" exitCode=0 Nov 03 19:24:26.166459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.166291 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" Nov 03 19:24:26.166459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.166315 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" event={"ID":"d7085f31-a663-487c-8683-a3ddc958e1de","Type":"ContainerDied","Data":"ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb"} Nov 03 19:24:26.166459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.166382 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx" event={"ID":"d7085f31-a663-487c-8683-a3ddc958e1de","Type":"ContainerDied","Data":"ed8f3c2891b62fe97066d83e387efd7cf236fbfd309329f1d0dc940dec1d2749"} Nov 03 19:24:26.166459 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.166405 4948 scope.go:122] "RemoveContainer" containerID="ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb" Nov 03 19:24:26.168248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.168219 4948 generic.go:356] "Generic (PLEG): container finished" podID="c28d4dd5-c3df-4b8f-918c-527d662015da" containerID="5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265" exitCode=0 Nov 03 19:24:26.168389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.168268 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" event={"ID":"c28d4dd5-c3df-4b8f-918c-527d662015da","Type":"ContainerDied","Data":"5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265"} Nov 03 19:24:26.168389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.168283 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" Nov 03 19:24:26.168389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.168309 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc" event={"ID":"c28d4dd5-c3df-4b8f-918c-527d662015da","Type":"ContainerDied","Data":"9ad7275e83c9c39a01f94afcce3913d16083a5479254996b8f72432be3c811a8"} Nov 03 19:24:26.183322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.183295 4948 scope.go:122] "RemoveContainer" containerID="ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb" Nov 03 19:24:26.183610 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:26.183591 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb\": container with ID starting with ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb not found: ID does not exist" containerID="ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb" Nov 03 19:24:26.183712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.183619 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb"} err="failed to get container status \"ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb\": rpc error: code = NotFound desc = could not find container \"ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb\": container with ID starting with ea3f7cf8ef0bf24d510dc468d6f54c094d76e792a98b1f7f75a31c9fac70a1eb not found: ID does not exist" Nov 03 19:24:26.183712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.183635 4948 scope.go:122] "RemoveContainer" containerID="5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265" Nov 03 19:24:26.193453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.193431 4948 scope.go:122] "RemoveContainer" containerID="5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265" Nov 03 19:24:26.193764 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:26.193740 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265\": container with ID starting with 5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265 not found: ID does not exist" containerID="5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265" Nov 03 19:24:26.193870 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.193769 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265"} err="failed to get container status \"5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265\": rpc error: code = NotFound desc = could not find container \"5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265\": container with ID starting with 5e854f8a6d18f1baaad98426e26ed432f0b23a804f8dd4966147006959755265 not found: ID does not exist" Nov 03 19:24:26.325537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.325484 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:24:26.335117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.335088 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx"] Nov 03 19:24:26.340793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.340769 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-lfpcx"] Nov 03 19:24:26.425690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.423955 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7298fafe-3b44-4a92-83a6-461bf774367d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7298fafe-3b44-4a92-83a6-461bf774367d-openshift-service-ca\") pod \"7298fafe-3b44-4a92-83a6-461bf774367d\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " Nov 03 19:24:26.425690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.424754 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7298fafe-3b44-4a92-83a6-461bf774367d-kube-api-access-r6lvz\" (UniqueName: \"kubernetes.io/projected/7298fafe-3b44-4a92-83a6-461bf774367d-kube-api-access-r6lvz\") pod \"7298fafe-3b44-4a92-83a6-461bf774367d\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " Nov 03 19:24:26.425690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.424886 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls\") pod \"7298fafe-3b44-4a92-83a6-461bf774367d\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " Nov 03 19:24:26.425690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.424992 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-metrics-certs\") pod \"7298fafe-3b44-4a92-83a6-461bf774367d\" (UID: \"7298fafe-3b44-4a92-83a6-461bf774367d\") " Nov 03 19:24:26.434906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.427308 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7298fafe-3b44-4a92-83a6-461bf774367d-openshift-service-ca" pod "7298fafe-3b44-4a92-83a6-461bf774367d" (UID: "7298fafe-3b44-4a92-83a6-461bf774367d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:26.434906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.434173 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7298fafe-3b44-4a92-83a6-461bf774367d-kube-api-access-r6lvz" pod "7298fafe-3b44-4a92-83a6-461bf774367d" (UID: "7298fafe-3b44-4a92-83a6-461bf774367d"). InnerVolumeSpecName "kube-api-access-r6lvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:26.435672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.435630 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-metrics-certs" pod "7298fafe-3b44-4a92-83a6-461bf774367d" (UID: "7298fafe-3b44-4a92-83a6-461bf774367d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:26.438588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.438549 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls" pod "7298fafe-3b44-4a92-83a6-461bf774367d" (UID: "7298fafe-3b44-4a92-83a6-461bf774367d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:26.451735 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.451670 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc"] Nov 03 19:24:26.457453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.457420 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d74f801-d4cf58969-zjtsc"] Nov 03 19:24:26.531468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.530980 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:26.531468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.531022 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7298fafe-3b44-4a92-83a6-461bf774367d-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:26.531468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.531039 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lvz\" (UniqueName: \"kubernetes.io/projected/7298fafe-3b44-4a92-83a6-461bf774367d-kube-api-access-r6lvz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:26.531468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.531057 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7298fafe-3b44-4a92-83a6-461bf774367d-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:26.754628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.754588 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28d4dd5-c3df-4b8f-918c-527d662015da" path="/var/lib/kubelet/pods/c28d4dd5-c3df-4b8f-918c-527d662015da/volumes" Nov 03 19:24:26.755198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:26.755174 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7085f31-a663-487c-8683-a3ddc958e1de" path="/var/lib/kubelet/pods/d7085f31-a663-487c-8683-a3ddc958e1de/volumes" Nov 03 19:24:27.180014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.179980 4948 generic.go:356] "Generic (PLEG): container finished" podID="7298fafe-3b44-4a92-83a6-461bf774367d" containerID="0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00" exitCode=0 Nov 03 19:24:27.180233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.180045 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" event={"ID":"7298fafe-3b44-4a92-83a6-461bf774367d","Type":"ContainerDied","Data":"0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00"} Nov 03 19:24:27.180233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.180087 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" event={"ID":"7298fafe-3b44-4a92-83a6-461bf774367d","Type":"ContainerDied","Data":"794c27f6f8ad5d53e0217d1f5f163e53c4f43a899e7f2aa33571ae20903ea165"} Nov 03 19:24:27.180233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.180109 4948 scope.go:122] "RemoveContainer" containerID="0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00" Nov 03 19:24:27.180233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.180048 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv" Nov 03 19:24:27.193284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.193258 4948 scope.go:122] "RemoveContainer" containerID="0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00" Nov 03 19:24:27.193588 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:27.193565 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00\": container with ID starting with 0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00 not found: ID does not exist" containerID="0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00" Nov 03 19:24:27.193655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.193598 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00"} err="failed to get container status \"0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00\": rpc error: code = NotFound desc = could not find container \"0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00\": container with ID starting with 0e2c3f5bba6442ba8d8edc16ab292df2e7a811d28d6e5bf57d0d019cac691d00 not found: ID does not exist" Nov 03 19:24:27.349028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.348995 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv"] Nov 03 19:24:27.400151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:27.400112 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d74f801-5fbdb695cf-vxjbv"] Nov 03 19:24:28.753624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:28.753591 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7298fafe-3b44-4a92-83a6-461bf774367d" path="/var/lib/kubelet/pods/7298fafe-3b44-4a92-83a6-461bf774367d/volumes" Nov 03 19:24:32.794178 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:32.794134 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:24:41.188845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.188800 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr"] Nov 03 19:24:41.189254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.189108 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" podUID="47c04ffd-044b-4499-ac87-71d9fd5d238a" containerName="maas-api" containerID="cri-o://35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7" gracePeriod=30 Nov 03 19:24:41.470490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.470464 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:24:41.601574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.601536 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c04ffd-044b-4499-ac87-71d9fd5d238a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c04ffd-044b-4499-ac87-71d9fd5d238a-openshift-service-ca\") pod \"47c04ffd-044b-4499-ac87-71d9fd5d238a\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " Nov 03 19:24:41.601793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.601601 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-maas-api-tls\") pod \"47c04ffd-044b-4499-ac87-71d9fd5d238a\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " Nov 03 19:24:41.601793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.601641 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c04ffd-044b-4499-ac87-71d9fd5d238a-kube-api-access-mxqcn\" (UniqueName: \"kubernetes.io/projected/47c04ffd-044b-4499-ac87-71d9fd5d238a-kube-api-access-mxqcn\") pod \"47c04ffd-044b-4499-ac87-71d9fd5d238a\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " Nov 03 19:24:41.601793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.601787 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-metrics-certs\") pod \"47c04ffd-044b-4499-ac87-71d9fd5d238a\" (UID: \"47c04ffd-044b-4499-ac87-71d9fd5d238a\") " Nov 03 19:24:41.602001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.601912 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c04ffd-044b-4499-ac87-71d9fd5d238a-openshift-service-ca" pod "47c04ffd-044b-4499-ac87-71d9fd5d238a" (UID: "47c04ffd-044b-4499-ac87-71d9fd5d238a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:41.602257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.602232 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c04ffd-044b-4499-ac87-71d9fd5d238a-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:41.603996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.603966 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-metrics-certs" pod "47c04ffd-044b-4499-ac87-71d9fd5d238a" (UID: "47c04ffd-044b-4499-ac87-71d9fd5d238a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:41.604197 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.604178 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c04ffd-044b-4499-ac87-71d9fd5d238a-kube-api-access-mxqcn" pod "47c04ffd-044b-4499-ac87-71d9fd5d238a" (UID: "47c04ffd-044b-4499-ac87-71d9fd5d238a"). InnerVolumeSpecName "kube-api-access-mxqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:41.604382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.604362 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-maas-api-tls" pod "47c04ffd-044b-4499-ac87-71d9fd5d238a" (UID: "47c04ffd-044b-4499-ac87-71d9fd5d238a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:41.712613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.712513 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:41.712613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.712542 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c04ffd-044b-4499-ac87-71d9fd5d238a-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:41.712613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:41.712556 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxqcn\" (UniqueName: \"kubernetes.io/projected/47c04ffd-044b-4499-ac87-71d9fd5d238a-kube-api-access-mxqcn\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:42.290782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.290744 4948 generic.go:356] "Generic (PLEG): container finished" podID="47c04ffd-044b-4499-ac87-71d9fd5d238a" containerID="35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7" exitCode=0 Nov 03 19:24:42.291205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.290800 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" Nov 03 19:24:42.291205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.290832 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" event={"ID":"47c04ffd-044b-4499-ac87-71d9fd5d238a","Type":"ContainerDied","Data":"35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7"} Nov 03 19:24:42.291205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.290894 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr" event={"ID":"47c04ffd-044b-4499-ac87-71d9fd5d238a","Type":"ContainerDied","Data":"3d6f6bb7fbc93af49a3289f7bd288f458cd0b2407365478a476300d8014f9e1d"} Nov 03 19:24:42.291205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.290924 4948 scope.go:122] "RemoveContainer" containerID="35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7" Nov 03 19:24:42.303648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.303629 4948 scope.go:122] "RemoveContainer" containerID="35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7" Nov 03 19:24:42.304023 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:24:42.304002 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7\": container with ID starting with 35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7 not found: ID does not exist" containerID="35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7" Nov 03 19:24:42.304076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.304031 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7"} err="failed to get container status \"35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7\": rpc error: code = NotFound desc = could not find container \"35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7\": container with ID starting with 35ad09524199fb816c964ec4afbfe8aba8e7b1f4ecf1c2015c20642845f5a6b7 not found: ID does not exist" Nov 03 19:24:42.462785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.462685 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr"] Nov 03 19:24:42.467081 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.467057 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9dd9f2-7988bb5586-s5rcr"] Nov 03 19:24:42.753244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:42.753155 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c04ffd-044b-4499-ac87-71d9fd5d238a" path="/var/lib/kubelet/pods/47c04ffd-044b-4499-ac87-71d9fd5d238a/volumes" Nov 03 19:24:54.400691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:54.400646 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p"] Nov 03 19:24:54.401340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:54.400906 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" containerID="cri-o://27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb" gracePeriod=30 Nov 03 19:24:54.451745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:54.451683 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f"] Nov 03 19:24:54.451998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:54.451953 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" containerID="cri-o://f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76" gracePeriod=30 Nov 03 19:24:58.548071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:58.548034 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4"] Nov 03 19:24:58.548572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:58.548290 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" containerID="cri-o://5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530" gracePeriod=30 Nov 03 19:24:59.729241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.729211 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:24:59.756377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.756349 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:24:59.842029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.841993 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-kube-api-access-z5nfc\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-kube-api-access-z5nfc\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.842225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842045 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-certs\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.842225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842121 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-socket\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.842225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842163 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-envoy\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.842225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842194 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-certs\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.842456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842246 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-envoy\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.842456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842334 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-socket\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.842456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842376 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-credential-socket\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.842456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842387 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-certs" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.842456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842419 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-podinfo\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.842456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842423 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-socket" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.842830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842456 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-data\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.842830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842497 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa9a816-c24a-49a1-bdda-20144fa2d008-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9a816-c24a-49a1-bdda-20144fa2d008-istiod-ca-cert\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.842830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842527 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-kube-api-access-lzwk8\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-kube-api-access-lzwk8\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.842830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842575 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istiod-ca-cert\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.842830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842622 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-credential-socket\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.842830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842665 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-credential-socket" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.843174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.842900 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-credential-socket" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.843174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843111 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-certs" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.843174 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843164 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-token\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.843342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843212 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-podinfo\") pod \"8aa9a816-c24a-49a1-bdda-20144fa2d008\" (UID: \"8aa9a816-c24a-49a1-bdda-20144fa2d008\") " Nov 03 19:24:59.843342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843226 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istiod-ca-cert" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:59.843457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843339 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-data" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.843457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843246 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-data\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.843457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843411 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-token\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-token\") pod \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\" (UID: \"49b28c18-efa0-4d8f-8b3f-92a2ed348058\") " Nov 03 19:24:59.843630 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843518 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa9a816-c24a-49a1-bdda-20144fa2d008-istiod-ca-cert" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:59.843869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843753 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-socket" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.843869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843771 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-data" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.844041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.843984 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844006 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844022 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844041 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844038 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844051 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9a816-c24a-49a1-bdda-20144fa2d008-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844065 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844082 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844097 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844110 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844125 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.844936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844906 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-kube-api-access-z5nfc" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "kube-api-access-z5nfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:59.845075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.844952 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-envoy" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.845399 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.845368 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-kube-api-access-lzwk8" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "kube-api-access-lzwk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:59.845996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.845953 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-token" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:59.845996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.845946 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-envoy" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:59.846199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.846176 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-podinfo" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:59.846390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.846372 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-podinfo" pod "8aa9a816-c24a-49a1-bdda-20144fa2d008" (UID: "8aa9a816-c24a-49a1-bdda-20144fa2d008"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:59.846782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.846766 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-token" pod "49b28c18-efa0-4d8f-8b3f-92a2ed348058" (UID: "49b28c18-efa0-4d8f-8b3f-92a2ed348058"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:59.945473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945434 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.945473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945468 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwk8\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-kube-api-access-lzwk8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.945473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945480 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.945846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945489 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.945846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945497 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.945846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945508 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nfc\" (UniqueName: \"kubernetes.io/projected/8aa9a816-c24a-49a1-bdda-20144fa2d008-kube-api-access-z5nfc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.945846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945520 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b28c18-efa0-4d8f-8b3f-92a2ed348058-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:59.945846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:24:59.945533 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9a816-c24a-49a1-bdda-20144fa2d008-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:00.421419 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.421378 4948 generic.go:356] "Generic (PLEG): container finished" podID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerID="27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb" exitCode=0 Nov 03 19:25:00.421613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.421429 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" Nov 03 19:25:00.421613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.421460 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" event={"ID":"8aa9a816-c24a-49a1-bdda-20144fa2d008","Type":"ContainerDied","Data":"27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb"} Nov 03 19:25:00.421613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.421521 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p" event={"ID":"8aa9a816-c24a-49a1-bdda-20144fa2d008","Type":"ContainerDied","Data":"c5cc1af9fb6adbed1a46a6388e7a3eff001fac582a2a1c02b8671e9f91264232"} Nov 03 19:25:00.421613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.421554 4948 scope.go:122] "RemoveContainer" containerID="27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb" Nov 03 19:25:00.423385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.423359 4948 generic.go:356] "Generic (PLEG): container finished" podID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerID="f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76" exitCode=0 Nov 03 19:25:00.423532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.423414 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" event={"ID":"49b28c18-efa0-4d8f-8b3f-92a2ed348058","Type":"ContainerDied","Data":"f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76"} Nov 03 19:25:00.423532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.423417 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" Nov 03 19:25:00.423532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.423450 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f" event={"ID":"49b28c18-efa0-4d8f-8b3f-92a2ed348058","Type":"ContainerDied","Data":"4fb6e030067625dc1cb12cdc5e11b9e3a1bc8decf4bdf72d4dea39386d53039b"} Nov 03 19:25:00.444088 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.444065 4948 scope.go:122] "RemoveContainer" containerID="27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb" Nov 03 19:25:00.444413 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:00.444394 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb\": container with ID starting with 27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb not found: ID does not exist" containerID="27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb" Nov 03 19:25:00.444473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.444423 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb"} err="failed to get container status \"27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb\": rpc error: code = NotFound desc = could not find container \"27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb\": container with ID starting with 27f6ad4449d58d460f2aee4a9f4d254a27b882593a1a0b0bf7220de1e40d4edb not found: ID does not exist" Nov 03 19:25:00.444473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.444440 4948 scope.go:122] "RemoveContainer" containerID="f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76" Nov 03 19:25:00.454186 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.454167 4948 scope.go:122] "RemoveContainer" containerID="f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76" Nov 03 19:25:00.454436 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:00.454419 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76\": container with ID starting with f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76 not found: ID does not exist" containerID="f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76" Nov 03 19:25:00.454500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.454442 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76"} err="failed to get container status \"f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76\": rpc error: code = NotFound desc = could not find container \"f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76\": container with ID starting with f3fc8156ab7b1e4ab2e4ec2095ae23a59e52553bd7bad4801945c6ff6c54cd76 not found: ID does not exist" Nov 03 19:25:00.590839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.590801 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f"] Nov 03 19:25:00.649980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.644945 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8n78f"] Nov 03 19:25:00.652967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.652938 4948 scope.go:122] "RemoveContainer" containerID="cfa0e66b2435d8bc57cb27a71f7ee9ce5b63c625d55a64c15898d8c306ee03e4" Nov 03 19:25:00.663105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.663080 4948 scope.go:122] "RemoveContainer" containerID="880f8dc4648e16d4a972a6f2744d19bcdf6f516bf7a86c9f6505a3b1874b1fd4" Nov 03 19:25:00.672888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.672867 4948 scope.go:122] "RemoveContainer" containerID="64265189dd865cc25674b65606e9682e7dd7066e28a4c0efbabe4e222b9439fc" Nov 03 19:25:00.681730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.681681 4948 scope.go:122] "RemoveContainer" containerID="8ab3e52328cd4a1643d1dd58d1e1c92fc7367a6dc1dbc2cc4464746ad54d2321" Nov 03 19:25:00.690526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.690481 4948 scope.go:122] "RemoveContainer" containerID="69585d98fb69d13f48064197385e009306a94a4140483f492fb1372f57aa295b" Nov 03 19:25:00.707713 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.707677 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p"] Nov 03 19:25:00.712569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.712544 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-npq2p"] Nov 03 19:25:00.753983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.753957 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" path="/var/lib/kubelet/pods/49b28c18-efa0-4d8f-8b3f-92a2ed348058/volumes" Nov 03 19:25:00.754650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:00.754628 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" path="/var/lib/kubelet/pods/8aa9a816-c24a-49a1-bdda-20144fa2d008/volumes" Nov 03 19:25:02.683905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:02.683870 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd"] Nov 03 19:25:02.796594 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:02.796556 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:25:03.821052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.820990 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 19:25:03.887981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.887945 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istiod-ca-cert\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888008 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-envoy\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888080 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-data\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888120 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-kube-api-access-cwxjj\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-kube-api-access-cwxjj\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888163 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-certs\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888196 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-socket\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888268 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-credential-socket\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888328 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-token\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-token\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888346 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istiod-ca-cert" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:03.888415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888354 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-podinfo\") pod \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\" (UID: \"26d25ebd-3ae1-409d-9f84-70bb798ba6a8\") " Nov 03 19:25:03.888664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888608 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-socket" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:03.888664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888629 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-credential-socket" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:03.889478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.888966 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-certs" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:03.889478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.889161 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.889478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.889183 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.889478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.889199 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.889478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.889213 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.889907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.889739 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-data" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:03.890940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.890905 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-envoy" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:03.891042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.890970 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-podinfo" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:25:03.891042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.891014 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-kube-api-access-cwxjj" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "kube-api-access-cwxjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:03.891325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.891306 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-token" pod "26d25ebd-3ae1-409d-9f84-70bb798ba6a8" (UID: "26d25ebd-3ae1-409d-9f84-70bb798ba6a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:03.991039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.991000 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.991039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.991035 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwxjj\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-kube-api-access-cwxjj\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.991270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.991050 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.991270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.991063 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:03.991270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:03.991076 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d25ebd-3ae1-409d-9f84-70bb798ba6a8-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:04.451579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.451548 4948 generic.go:356] "Generic (PLEG): container finished" podID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerID="5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530" exitCode=0 Nov 03 19:25:04.451813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.451605 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" Nov 03 19:25:04.451813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.451629 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" event={"ID":"26d25ebd-3ae1-409d-9f84-70bb798ba6a8","Type":"ContainerDied","Data":"5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530"} Nov 03 19:25:04.451813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.451692 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4" event={"ID":"26d25ebd-3ae1-409d-9f84-70bb798ba6a8","Type":"ContainerDied","Data":"196aa90b76c1eedf82c5a936aa5766fcabc657e6c574c2450dd3863d65ddf985"} Nov 03 19:25:04.451813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.451734 4948 scope.go:122] "RemoveContainer" containerID="5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530" Nov 03 19:25:04.463137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.463118 4948 scope.go:122] "RemoveContainer" containerID="490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc" Nov 03 19:25:04.472484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.472467 4948 scope.go:122] "RemoveContainer" containerID="5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530" Nov 03 19:25:04.472793 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:04.472770 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530\": container with ID starting with 5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530 not found: ID does not exist" containerID="5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530" Nov 03 19:25:04.472846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.472802 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530"} err="failed to get container status \"5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530\": rpc error: code = NotFound desc = could not find container \"5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530\": container with ID starting with 5b8353792ccf859147ba905b9879652e5b88c55ac50c6e773f4f7b669871f530 not found: ID does not exist" Nov 03 19:25:04.472846 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.472823 4948 scope.go:122] "RemoveContainer" containerID="490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc" Nov 03 19:25:04.473087 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:04.473069 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc\": container with ID starting with 490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc not found: ID does not exist" containerID="490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc" Nov 03 19:25:04.473134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.473095 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc"} err="failed to get container status \"490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc\": rpc error: code = NotFound desc = could not find container \"490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc\": container with ID starting with 490914bdcb029ef3403b4ab72707be0e92ddcf40c00c868057883d0245ab08cc not found: ID does not exist" Nov 03 19:25:04.621231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.621192 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4"] Nov 03 19:25:04.672341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.672307 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-hsnb4"] Nov 03 19:25:04.753018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:04.752937 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" path="/var/lib/kubelet/pods/26d25ebd-3ae1-409d-9f84-70bb798ba6a8/volumes" Nov 03 19:25:07.684571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:07.684525 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" podUID="b8d74572-40d3-4a32-a5af-91349b852192" containerName="payload-pre-processing" containerID="cri-o://d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334" gracePeriod=25 Nov 03 19:25:10.121676 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.121641 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn"] Nov 03 19:25:10.122124 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.121876 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" podUID="cdabd8f8-e278-4673-bebf-10c6d30259ab" containerName="maas-api" containerID="cri-o://6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f" gracePeriod=30 Nov 03 19:25:10.405940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.405912 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:25:10.500300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.500261 4948 generic.go:356] "Generic (PLEG): container finished" podID="cdabd8f8-e278-4673-bebf-10c6d30259ab" containerID="6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f" exitCode=0 Nov 03 19:25:10.500498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.500350 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" Nov 03 19:25:10.500498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.500341 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" event={"ID":"cdabd8f8-e278-4673-bebf-10c6d30259ab","Type":"ContainerDied","Data":"6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f"} Nov 03 19:25:10.500606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.500502 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn" event={"ID":"cdabd8f8-e278-4673-bebf-10c6d30259ab","Type":"ContainerDied","Data":"91108ea27befcda785c8588518bdf25c7247eb03aa16df521075a85b43b54cfc"} Nov 03 19:25:10.500606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.500525 4948 scope.go:122] "RemoveContainer" containerID="6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f" Nov 03 19:25:10.521916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.521885 4948 scope.go:122] "RemoveContainer" containerID="6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f" Nov 03 19:25:10.528809 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:10.525106 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f\": container with ID starting with 6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f not found: ID does not exist" containerID="6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f" Nov 03 19:25:10.528809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.525179 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f"} err="failed to get container status \"6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f\": rpc error: code = NotFound desc = could not find container \"6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f\": container with ID starting with 6d6a2899634ef383c3570d57439b3780f410d25ead412cb404477b3cb429d21f not found: ID does not exist" Nov 03 19:25:10.571595 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.571561 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-maas-api-tls\") pod \"cdabd8f8-e278-4673-bebf-10c6d30259ab\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " Nov 03 19:25:10.571854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.571693 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdabd8f8-e278-4673-bebf-10c6d30259ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdabd8f8-e278-4673-bebf-10c6d30259ab-openshift-service-ca\") pod \"cdabd8f8-e278-4673-bebf-10c6d30259ab\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " Nov 03 19:25:10.571854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.571801 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-metrics-certs\") pod \"cdabd8f8-e278-4673-bebf-10c6d30259ab\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " Nov 03 19:25:10.572006 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.571943 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdabd8f8-e278-4673-bebf-10c6d30259ab-kube-api-access-qwsml\" (UniqueName: \"kubernetes.io/projected/cdabd8f8-e278-4673-bebf-10c6d30259ab-kube-api-access-qwsml\") pod \"cdabd8f8-e278-4673-bebf-10c6d30259ab\" (UID: \"cdabd8f8-e278-4673-bebf-10c6d30259ab\") " Nov 03 19:25:10.572534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.572503 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdabd8f8-e278-4673-bebf-10c6d30259ab-openshift-service-ca" pod "cdabd8f8-e278-4673-bebf-10c6d30259ab" (UID: "cdabd8f8-e278-4673-bebf-10c6d30259ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:10.574486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.574453 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-maas-api-tls" pod "cdabd8f8-e278-4673-bebf-10c6d30259ab" (UID: "cdabd8f8-e278-4673-bebf-10c6d30259ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:10.574880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.574857 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdabd8f8-e278-4673-bebf-10c6d30259ab-kube-api-access-qwsml" pod "cdabd8f8-e278-4673-bebf-10c6d30259ab" (UID: "cdabd8f8-e278-4673-bebf-10c6d30259ab"). InnerVolumeSpecName "kube-api-access-qwsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:10.577170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.577121 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-metrics-certs" pod "cdabd8f8-e278-4673-bebf-10c6d30259ab" (UID: "cdabd8f8-e278-4673-bebf-10c6d30259ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:10.673432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.673354 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:10.673432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.673396 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwsml\" (UniqueName: \"kubernetes.io/projected/cdabd8f8-e278-4673-bebf-10c6d30259ab-kube-api-access-qwsml\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:10.673432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.673413 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdabd8f8-e278-4673-bebf-10c6d30259ab-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:10.673432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.673427 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdabd8f8-e278-4673-bebf-10c6d30259ab-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:10.921284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.921240 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn"] Nov 03 19:25:10.924256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:10.924190 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-942af8-57b88f6bc7-mlttn"] Nov 03 19:25:12.754134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:12.754100 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdabd8f8-e278-4673-bebf-10c6d30259ab" path="/var/lib/kubelet/pods/cdabd8f8-e278-4673-bebf-10c6d30259ab/volumes" Nov 03 19:25:14.728773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:14.728731 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" podUID="b8d74572-40d3-4a32-a5af-91349b852192" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:25:17.823622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:17.823590 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428"] Nov 03 19:25:17.824077 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:17.823805 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" containerID="cri-o://bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad" gracePeriod=30 Nov 03 19:25:18.816036 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816003 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz"] Nov 03 19:25:18.816497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816484 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7298fafe-3b44-4a92-83a6-461bf774367d" containerName="maas-api" Nov 03 19:25:18.816497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816497 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7298fafe-3b44-4a92-83a6-461bf774367d" containerName="maas-api" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816513 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c28d4dd5-c3df-4b8f-918c-527d662015da" containerName="payload-processing" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816519 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28d4dd5-c3df-4b8f-918c-527d662015da" containerName="payload-processing" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816529 4948 cpu_manager.go:414] "containerMap: removing container" podUID="d7085f31-a663-487c-8683-a3ddc958e1de" containerName="payload-pre-processing" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816535 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7085f31-a663-487c-8683-a3ddc958e1de" containerName="payload-pre-processing" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816548 4948 cpu_manager.go:414] "containerMap: removing container" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816553 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816564 4948 cpu_manager.go:414] "containerMap: removing container" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816571 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816581 4948 cpu_manager.go:414] "containerMap: removing container" podUID="cdabd8f8-e278-4673-bebf-10c6d30259ab" containerName="maas-api" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816585 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdabd8f8-e278-4673-bebf-10c6d30259ab" containerName="maas-api" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816599 4948 cpu_manager.go:414] "containerMap: removing container" podUID="47c04ffd-044b-4499-ac87-71d9fd5d238a" containerName="maas-api" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816605 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c04ffd-044b-4499-ac87-71d9fd5d238a" containerName="maas-api" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816612 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" containerName="test-curl-7315-643cff" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816617 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" containerName="test-curl-7315-643cff" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816624 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" Nov 03 19:25:18.816624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816628 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816722 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c04ffd-044b-4499-ac87-71d9fd5d238a" containerName="maas-api" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816731 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8752d7c3-c9a1-42c9-b803-ace2eaa2aef6" containerName="test-curl-7315-643cff" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816736 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa9a816-c24a-49a1-bdda-20144fa2d008" containerName="istio-proxy" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816741 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816750 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7298fafe-3b44-4a92-83a6-461bf774367d" containerName="maas-api" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816757 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28d4dd5-c3df-4b8f-918c-527d662015da" containerName="payload-processing" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816762 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7085f31-a663-487c-8683-a3ddc958e1de" containerName="payload-pre-processing" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816768 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b28c18-efa0-4d8f-8b3f-92a2ed348058" containerName="istio-proxy" Nov 03 19:25:18.817173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.816773 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdabd8f8-e278-4673-bebf-10c6d30259ab" containerName="maas-api" Nov 03 19:25:18.820067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.820052 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.860847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.860817 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.861178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.860899 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.861178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.860959 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqt2\" (UniqueName: \"kubernetes.io/projected/86f4c2aa-ff06-4229-8427-6235312d96cb-kube-api-access-4qqt2\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.874180 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.874158 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:25:18.874307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.874209 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:25:18.877229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.877202 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz"] Nov 03 19:25:18.961628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.961595 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqt2\" (UniqueName: \"kubernetes.io/projected/86f4c2aa-ff06-4229-8427-6235312d96cb-kube-api-access-4qqt2\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.961847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.961728 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.961847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.961758 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.962369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.962342 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:18.964154 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:18.964132 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:19.026811 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:19.026786 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqt2\" (UniqueName: \"kubernetes.io/projected/86f4c2aa-ff06-4229-8427-6235312d96cb-kube-api-access-4qqt2\") pod \"maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:19.133680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:19.133609 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:19.513210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:19.513171 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz"] Nov 03 19:25:19.526951 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:25:19.526920 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f4c2aa_ff06_4229_8427_6235312d96cb.slice/crio-4189800c96d70ffc590735241007246237060383e6637b9c0f6177946b91da04 WatchSource:0}: Error finding container 4189800c96d70ffc590735241007246237060383e6637b9c0f6177946b91da04: Status 404 returned error can't find the container with id 4189800c96d70ffc590735241007246237060383e6637b9c0f6177946b91da04 Nov 03 19:25:19.563383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:19.563353 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" event={"ID":"86f4c2aa-ff06-4229-8427-6235312d96cb","Type":"ContainerStarted","Data":"4189800c96d70ffc590735241007246237060383e6637b9c0f6177946b91da04"} Nov 03 19:25:20.572110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:20.572075 4948 generic.go:356] "Generic (PLEG): container finished" podID="86f4c2aa-ff06-4229-8427-6235312d96cb" containerID="a1ca6363febeff88476e20fe9d430678f3deb9c49e62a1716e1206cf333aa6e6" exitCode=0 Nov 03 19:25:20.572485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:20.572176 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" event={"ID":"86f4c2aa-ff06-4229-8427-6235312d96cb","Type":"ContainerDied","Data":"a1ca6363febeff88476e20fe9d430678f3deb9c49e62a1716e1206cf333aa6e6"} Nov 03 19:25:21.742802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.742772 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:21.792685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.792647 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-openshift-ca\") pod \"86f4c2aa-ff06-4229-8427-6235312d96cb\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " Nov 03 19:25:21.792885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.792760 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-xks-ca\") pod \"86f4c2aa-ff06-4229-8427-6235312d96cb\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " Nov 03 19:25:21.792885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.792833 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f4c2aa-ff06-4229-8427-6235312d96cb-kube-api-access-4qqt2\" (UniqueName: \"kubernetes.io/projected/86f4c2aa-ff06-4229-8427-6235312d96cb-kube-api-access-4qqt2\") pod \"86f4c2aa-ff06-4229-8427-6235312d96cb\" (UID: \"86f4c2aa-ff06-4229-8427-6235312d96cb\") " Nov 03 19:25:21.793084 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.793059 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-openshift-ca" pod "86f4c2aa-ff06-4229-8427-6235312d96cb" (UID: "86f4c2aa-ff06-4229-8427-6235312d96cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:21.793299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.793275 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:21.794979 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.794951 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-xks-ca" pod "86f4c2aa-ff06-4229-8427-6235312d96cb" (UID: "86f4c2aa-ff06-4229-8427-6235312d96cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:21.795073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.795009 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f4c2aa-ff06-4229-8427-6235312d96cb-kube-api-access-4qqt2" pod "86f4c2aa-ff06-4229-8427-6235312d96cb" (UID: "86f4c2aa-ff06-4229-8427-6235312d96cb"). InnerVolumeSpecName "kube-api-access-4qqt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:21.894556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.894469 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqt2\" (UniqueName: \"kubernetes.io/projected/86f4c2aa-ff06-4229-8427-6235312d96cb-kube-api-access-4qqt2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:21.894556 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:21.894502 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4c2aa-ff06-4229-8427-6235312d96cb-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:22.586103 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:22.586075 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" Nov 03 19:25:22.586261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:22.586071 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz" event={"ID":"86f4c2aa-ff06-4229-8427-6235312d96cb","Type":"ContainerDied","Data":"4189800c96d70ffc590735241007246237060383e6637b9c0f6177946b91da04"} Nov 03 19:25:22.586261 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:22.586200 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4189800c96d70ffc590735241007246237060383e6637b9c0f6177946b91da04" Nov 03 19:25:23.084823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.084799 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:25:23.207685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.207656 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-credential-socket\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.207882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.207727 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-socket\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.207882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.207754 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-envoy\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.207882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.207801 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42791c9a-d4ed-4055-b330-f565ef971dc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42791c9a-d4ed-4055-b330-f565ef971dc7-istiod-ca-cert\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.207882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.207852 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-data\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.208790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.207893 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-podinfo\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.208790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.207924 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-certs\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.208790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.208002 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-socket" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:23.208790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.208346 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-certs" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:23.208790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.208430 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42791c9a-d4ed-4055-b330-f565ef971dc7-istiod-ca-cert" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:23.208790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.208677 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-data" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:23.209159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.208447 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-credential-socket" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:23.209159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.208995 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-kube-api-access-vprll\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-kube-api-access-vprll\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.209159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.209087 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-token\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-token\") pod \"42791c9a-d4ed-4055-b330-f565ef971dc7\" (UID: \"42791c9a-d4ed-4055-b330-f565ef971dc7\") " Nov 03 19:25:23.209634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.209509 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.209634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.209535 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.209634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.209551 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42791c9a-d4ed-4055-b330-f565ef971dc7-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.209634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.209568 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.209634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.209583 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.211484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.211322 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-envoy" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:23.211869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.211843 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-token" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:23.213554 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.213527 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-podinfo" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:25:23.213872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.213854 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-kube-api-access-vprll" pod "42791c9a-d4ed-4055-b330-f565ef971dc7" (UID: "42791c9a-d4ed-4055-b330-f565ef971dc7"). InnerVolumeSpecName "kube-api-access-vprll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:23.310233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.310197 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprll\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-kube-api-access-vprll\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.310233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.310230 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.310431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.310241 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.310431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.310252 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42791c9a-d4ed-4055-b330-f565ef971dc7-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:23.594297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.594204 4948 generic.go:356] "Generic (PLEG): container finished" podID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerID="bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad" exitCode=0 Nov 03 19:25:23.594297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.594267 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" Nov 03 19:25:23.594297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.594281 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" event={"ID":"42791c9a-d4ed-4055-b330-f565ef971dc7","Type":"ContainerDied","Data":"bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad"} Nov 03 19:25:23.594549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.594353 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428" event={"ID":"42791c9a-d4ed-4055-b330-f565ef971dc7","Type":"ContainerDied","Data":"e6831a8cd029673344e2ccdde2debc551d2229f1c10c64271c579fa2c87ceb10"} Nov 03 19:25:23.594549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.594384 4948 scope.go:122] "RemoveContainer" containerID="bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad" Nov 03 19:25:23.605526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.605506 4948 scope.go:122] "RemoveContainer" containerID="49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e" Nov 03 19:25:23.614839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.614820 4948 scope.go:122] "RemoveContainer" containerID="bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad" Nov 03 19:25:23.615084 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:23.615063 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad\": container with ID starting with bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad not found: ID does not exist" containerID="bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad" Nov 03 19:25:23.615150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.615093 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad"} err="failed to get container status \"bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad\": rpc error: code = NotFound desc = could not find container \"bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad\": container with ID starting with bf7c297955aea4a7792a7eedb9faa5f34514462d140aa99e5dac19146e4f02ad not found: ID does not exist" Nov 03 19:25:23.615150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.615114 4948 scope.go:122] "RemoveContainer" containerID="49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e" Nov 03 19:25:23.615370 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:23.615354 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e\": container with ID starting with 49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e not found: ID does not exist" containerID="49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e" Nov 03 19:25:23.615414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.615373 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e"} err="failed to get container status \"49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e\": rpc error: code = NotFound desc = could not find container \"49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e\": container with ID starting with 49a6c21656c80dc7962cf47166e47b0c23fafe352204c3d5432c58de8328c28e not found: ID does not exist" Nov 03 19:25:23.768042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.768010 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428"] Nov 03 19:25:23.818901 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:23.818868 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d74f801-openshift-default-78ddf467c-kv428"] Nov 03 19:25:24.729442 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:24.729403 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" podUID="b8d74572-40d3-4a32-a5af-91349b852192" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:25:24.753393 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:24.753363 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" path="/var/lib/kubelet/pods/42791c9a-d4ed-4055-b330-f565ef971dc7/volumes" Nov 03 19:25:27.954804 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:27.954742 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:25:28.059057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.059021 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d74572-40d3-4a32-a5af-91349b852192-kube-api-access-jvz8k\" (UniqueName: \"kubernetes.io/projected/b8d74572-40d3-4a32-a5af-91349b852192-kube-api-access-jvz8k\") pod \"b8d74572-40d3-4a32-a5af-91349b852192\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " Nov 03 19:25:28.059255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.059154 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d74572-40d3-4a32-a5af-91349b852192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d74572-40d3-4a32-a5af-91349b852192-plugins-config-volume\") pod \"b8d74572-40d3-4a32-a5af-91349b852192\" (UID: \"b8d74572-40d3-4a32-a5af-91349b852192\") " Nov 03 19:25:28.059542 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.059513 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d74572-40d3-4a32-a5af-91349b852192-plugins-config-volume" pod "b8d74572-40d3-4a32-a5af-91349b852192" (UID: "b8d74572-40d3-4a32-a5af-91349b852192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:28.061150 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.061118 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d74572-40d3-4a32-a5af-91349b852192-kube-api-access-jvz8k" pod "b8d74572-40d3-4a32-a5af-91349b852192" (UID: "b8d74572-40d3-4a32-a5af-91349b852192"). InnerVolumeSpecName "kube-api-access-jvz8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:28.160157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.160123 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvz8k\" (UniqueName: \"kubernetes.io/projected/b8d74572-40d3-4a32-a5af-91349b852192-kube-api-access-jvz8k\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:28.160157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.160153 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d74572-40d3-4a32-a5af-91349b852192-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:28.627469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.627428 4948 generic.go:356] "Generic (PLEG): container finished" podID="b8d74572-40d3-4a32-a5af-91349b852192" containerID="d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334" exitCode=0 Nov 03 19:25:28.627689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.627482 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" Nov 03 19:25:28.627689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.627520 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" event={"ID":"b8d74572-40d3-4a32-a5af-91349b852192","Type":"ContainerDied","Data":"d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334"} Nov 03 19:25:28.627689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.627592 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd" event={"ID":"b8d74572-40d3-4a32-a5af-91349b852192","Type":"ContainerDied","Data":"cb7c02687d62527d4a8e2d3f893b0bf3f5f8bea5e8295a93aee59f137d515ed3"} Nov 03 19:25:28.627689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.627624 4948 scope.go:122] "RemoveContainer" containerID="d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334" Nov 03 19:25:28.638489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.638469 4948 scope.go:122] "RemoveContainer" containerID="d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334" Nov 03 19:25:28.638820 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:28.638801 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334\": container with ID starting with d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334 not found: ID does not exist" containerID="d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334" Nov 03 19:25:28.638884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.638829 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334"} err="failed to get container status \"d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334\": rpc error: code = NotFound desc = could not find container \"d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334\": container with ID starting with d1ad2806f946605e7c54ae5b38ac43c23c0c98304e450ff8f9816e7eb32f2334 not found: ID does not exist" Nov 03 19:25:28.797802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.797768 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd"] Nov 03 19:25:28.847035 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:28.847001 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-942af8-6d9ff55467-fwgpd"] Nov 03 19:25:30.463523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:30.463304 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt"] Nov 03 19:25:30.755366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:30.755282 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d74572-40d3-4a32-a5af-91349b852192" path="/var/lib/kubelet/pods/b8d74572-40d3-4a32-a5af-91349b852192/volumes" Nov 03 19:25:32.792572 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:32.792532 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:25:32.889867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:32.889830 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v"] Nov 03 19:25:33.002487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:33.002453 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq"] Nov 03 19:25:35.464132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.464040 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" podUID="c9e06228-9483-46d1-937c-5b81dda76fb4" containerName="payload-pre-processing" containerID="cri-o://5cb78c0770d2fe98732e2edddb1d6d56759826954d25205064c3de239ba6171c" gracePeriod=25 Nov 03 19:25:35.677910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.677878 4948 generic.go:356] "Generic (PLEG): container finished" podID="c9e06228-9483-46d1-937c-5b81dda76fb4" containerID="5cb78c0770d2fe98732e2edddb1d6d56759826954d25205064c3de239ba6171c" exitCode=0 Nov 03 19:25:35.678092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.678042 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" event={"ID":"c9e06228-9483-46d1-937c-5b81dda76fb4","Type":"ContainerDied","Data":"5cb78c0770d2fe98732e2edddb1d6d56759826954d25205064c3de239ba6171c"} Nov 03 19:25:35.737802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.737773 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 19:25:35.845976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.845928 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e06228-9483-46d1-937c-5b81dda76fb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e06228-9483-46d1-937c-5b81dda76fb4-plugins-config-volume\") pod \"c9e06228-9483-46d1-937c-5b81dda76fb4\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " Nov 03 19:25:35.846512 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.846489 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e06228-9483-46d1-937c-5b81dda76fb4-kube-api-access-r5rk7\" (UniqueName: \"kubernetes.io/projected/c9e06228-9483-46d1-937c-5b81dda76fb4-kube-api-access-r5rk7\") pod \"c9e06228-9483-46d1-937c-5b81dda76fb4\" (UID: \"c9e06228-9483-46d1-937c-5b81dda76fb4\") " Nov 03 19:25:35.849799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.849715 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e06228-9483-46d1-937c-5b81dda76fb4-plugins-config-volume" pod "c9e06228-9483-46d1-937c-5b81dda76fb4" (UID: "c9e06228-9483-46d1-937c-5b81dda76fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:35.849799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.849778 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e06228-9483-46d1-937c-5b81dda76fb4-kube-api-access-r5rk7" pod "c9e06228-9483-46d1-937c-5b81dda76fb4" (UID: "c9e06228-9483-46d1-937c-5b81dda76fb4"). InnerVolumeSpecName "kube-api-access-r5rk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:35.947987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.947954 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5rk7\" (UniqueName: \"kubernetes.io/projected/c9e06228-9483-46d1-937c-5b81dda76fb4-kube-api-access-r5rk7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:35.947987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:35.947981 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e06228-9483-46d1-937c-5b81dda76fb4-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:36.686011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:36.685981 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" Nov 03 19:25:36.686452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:36.685985 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt" event={"ID":"c9e06228-9483-46d1-937c-5b81dda76fb4","Type":"ContainerDied","Data":"7c36ff5bd27d27295712255721a35cd7c97f0278725547ae30a37bb0df11ed9c"} Nov 03 19:25:36.686452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:36.686114 4948 scope.go:122] "RemoveContainer" containerID="5cb78c0770d2fe98732e2edddb1d6d56759826954d25205064c3de239ba6171c" Nov 03 19:25:36.855670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:36.855633 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt"] Nov 03 19:25:36.906731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:36.906674 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-355e1a-5c86d6b775-2kbjt"] Nov 03 19:25:37.890488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:37.890451 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" podUID="74868d4d-edf2-480a-b7b9-4aa75b7c667d" containerName="payload-processing" containerID="cri-o://5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9" gracePeriod=25 Nov 03 19:25:38.002967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.002930 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" podUID="fe5471ae-666f-44b6-a472-1cb21c9c4e0a" containerName="payload-pre-processing" containerID="cri-o://d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c" gracePeriod=25 Nov 03 19:25:38.148618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.148560 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:25:38.261052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.261023 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:25:38.272917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.272891 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74868d4d-edf2-480a-b7b9-4aa75b7c667d-kube-api-access-2fdn9\" (UniqueName: \"kubernetes.io/projected/74868d4d-edf2-480a-b7b9-4aa75b7c667d-kube-api-access-2fdn9\") pod \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " Nov 03 19:25:38.273346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.272945 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74868d4d-edf2-480a-b7b9-4aa75b7c667d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74868d4d-edf2-480a-b7b9-4aa75b7c667d-plugins-config-volume\") pod \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\" (UID: \"74868d4d-edf2-480a-b7b9-4aa75b7c667d\") " Nov 03 19:25:38.273346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.273301 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74868d4d-edf2-480a-b7b9-4aa75b7c667d-plugins-config-volume" pod "74868d4d-edf2-480a-b7b9-4aa75b7c667d" (UID: "74868d4d-edf2-480a-b7b9-4aa75b7c667d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:38.275085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.275062 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74868d4d-edf2-480a-b7b9-4aa75b7c667d-kube-api-access-2fdn9" pod "74868d4d-edf2-480a-b7b9-4aa75b7c667d" (UID: "74868d4d-edf2-480a-b7b9-4aa75b7c667d"). InnerVolumeSpecName "kube-api-access-2fdn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:38.373753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.373723 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-kube-api-access-z72bb\") pod \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " Nov 03 19:25:38.373963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.373768 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-plugins-config-volume\") pod \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\" (UID: \"fe5471ae-666f-44b6-a472-1cb21c9c4e0a\") " Nov 03 19:25:38.374080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.374064 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fdn9\" (UniqueName: \"kubernetes.io/projected/74868d4d-edf2-480a-b7b9-4aa75b7c667d-kube-api-access-2fdn9\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.374080 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.374079 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74868d4d-edf2-480a-b7b9-4aa75b7c667d-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.374228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.374200 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-plugins-config-volume" pod "fe5471ae-666f-44b6-a472-1cb21c9c4e0a" (UID: "fe5471ae-666f-44b6-a472-1cb21c9c4e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:38.375713 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.375680 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-kube-api-access-z72bb" pod "fe5471ae-666f-44b6-a472-1cb21c9c4e0a" (UID: "fe5471ae-666f-44b6-a472-1cb21c9c4e0a"). InnerVolumeSpecName "kube-api-access-z72bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:38.475504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.475465 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-kube-api-access-z72bb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.475504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.475496 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5471ae-666f-44b6-a472-1cb21c9c4e0a-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.703987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.703944 4948 generic.go:356] "Generic (PLEG): container finished" podID="fe5471ae-666f-44b6-a472-1cb21c9c4e0a" containerID="d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c" exitCode=0 Nov 03 19:25:38.704178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.703999 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" Nov 03 19:25:38.704178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.704104 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" event={"ID":"fe5471ae-666f-44b6-a472-1cb21c9c4e0a","Type":"ContainerDied","Data":"d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c"} Nov 03 19:25:38.704178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.704143 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq" event={"ID":"fe5471ae-666f-44b6-a472-1cb21c9c4e0a","Type":"ContainerDied","Data":"7345077d4e19be237202253263a0dc44394e729d7b3156a3c14079bd7b0c04e0"} Nov 03 19:25:38.704178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.704164 4948 scope.go:122] "RemoveContainer" containerID="d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c" Nov 03 19:25:38.712039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.707362 4948 generic.go:356] "Generic (PLEG): container finished" podID="74868d4d-edf2-480a-b7b9-4aa75b7c667d" containerID="5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9" exitCode=0 Nov 03 19:25:38.712039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.707413 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" Nov 03 19:25:38.712039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.707442 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" event={"ID":"74868d4d-edf2-480a-b7b9-4aa75b7c667d","Type":"ContainerDied","Data":"5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9"} Nov 03 19:25:38.712039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.707498 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v" event={"ID":"74868d4d-edf2-480a-b7b9-4aa75b7c667d","Type":"ContainerDied","Data":"4a7c626b4862ac1a27051d34acc84b565ff98fd7bb55471d794b232caf026a11"} Nov 03 19:25:38.723760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.722820 4948 scope.go:122] "RemoveContainer" containerID="d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c" Nov 03 19:25:38.724169 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:38.724143 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c\": container with ID starting with d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c not found: ID does not exist" containerID="d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c" Nov 03 19:25:38.724286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.724182 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c"} err="failed to get container status \"d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c\": rpc error: code = NotFound desc = could not find container \"d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c\": container with ID starting with d3b2bd11c56dd831979a3a9c33328e2a44b00e12d439652c2329b7ede8886b0c not found: ID does not exist" Nov 03 19:25:38.724286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.724208 4948 scope.go:122] "RemoveContainer" containerID="5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9" Nov 03 19:25:38.734739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.734716 4948 scope.go:122] "RemoveContainer" containerID="5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9" Nov 03 19:25:38.735127 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:25:38.735088 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9\": container with ID starting with 5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9 not found: ID does not exist" containerID="5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9" Nov 03 19:25:38.735255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.735125 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9"} err="failed to get container status \"5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9\": rpc error: code = NotFound desc = could not find container \"5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9\": container with ID starting with 5a776145b3f32f098e056dab72686c69f5d62aca64df59e8f283e9cebed4f9c9 not found: ID does not exist" Nov 03 19:25:38.754055 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.754032 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e06228-9483-46d1-937c-5b81dda76fb4" path="/var/lib/kubelet/pods/c9e06228-9483-46d1-937c-5b81dda76fb4/volumes" Nov 03 19:25:38.875039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.875001 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v"] Nov 03 19:25:38.879497 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.879469 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770e67-dbf74bfbf-vhp4v"] Nov 03 19:25:38.993809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.993691 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq"] Nov 03 19:25:38.999628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:38.999606 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770e67-85547c9dc-9mtrq"] Nov 03 19:25:40.753397 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.753357 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74868d4d-edf2-480a-b7b9-4aa75b7c667d" path="/var/lib/kubelet/pods/74868d4d-edf2-480a-b7b9-4aa75b7c667d/volumes" Nov 03 19:25:40.753889 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.753766 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5471ae-666f-44b6-a472-1cb21c9c4e0a" path="/var/lib/kubelet/pods/fe5471ae-666f-44b6-a472-1cb21c9c4e0a/volumes" Nov 03 19:25:40.832183 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832142 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp"] Nov 03 19:25:40.832671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832651 4948 cpu_manager.go:414] "containerMap: removing container" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832673 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832690 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b8d74572-40d3-4a32-a5af-91349b852192" containerName="payload-pre-processing" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832712 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d74572-40d3-4a32-a5af-91349b852192" containerName="payload-pre-processing" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832734 4948 cpu_manager.go:414] "containerMap: removing container" podUID="74868d4d-edf2-480a-b7b9-4aa75b7c667d" containerName="payload-processing" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832742 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74868d4d-edf2-480a-b7b9-4aa75b7c667d" containerName="payload-processing" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832764 4948 cpu_manager.go:414] "containerMap: removing container" podUID="86f4c2aa-ff06-4229-8427-6235312d96cb" containerName="revoke-keys" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832771 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f4c2aa-ff06-4229-8427-6235312d96cb" containerName="revoke-keys" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832781 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c9e06228-9483-46d1-937c-5b81dda76fb4" containerName="payload-pre-processing" Nov 03 19:25:40.832806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832789 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e06228-9483-46d1-937c-5b81dda76fb4" containerName="payload-pre-processing" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832810 4948 cpu_manager.go:414] "containerMap: removing container" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832818 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832842 4948 cpu_manager.go:414] "containerMap: removing container" podUID="fe5471ae-666f-44b6-a472-1cb21c9c4e0a" containerName="payload-pre-processing" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832851 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5471ae-666f-44b6-a472-1cb21c9c4e0a" containerName="payload-pre-processing" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.832996 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d74572-40d3-4a32-a5af-91349b852192" containerName="payload-pre-processing" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.833010 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="74868d4d-edf2-480a-b7b9-4aa75b7c667d" containerName="payload-processing" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.833022 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f4c2aa-ff06-4229-8427-6235312d96cb" containerName="revoke-keys" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.833032 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e06228-9483-46d1-937c-5b81dda76fb4" containerName="payload-pre-processing" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.833045 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d25ebd-3ae1-409d-9f84-70bb798ba6a8" containerName="istio-proxy" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.833053 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" Nov 03 19:25:40.833269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.833064 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5471ae-666f-44b6-a472-1cb21c9c4e0a" containerName="payload-pre-processing" Nov 03 19:25:40.838083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.838061 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:40.889965 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.889928 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp"] Nov 03 19:25:40.938286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.938254 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:25:40.938538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:40.938420 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:25:41.001146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.001114 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.001352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.001225 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.001352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.001311 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qlf\" (UniqueName: \"kubernetes.io/projected/5eb6bcdf-a951-49cb-ae86-180449f264c7-kube-api-access-z6qlf\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.102832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.102736 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.103017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.102843 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.103017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.102936 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qlf\" (UniqueName: \"kubernetes.io/projected/5eb6bcdf-a951-49cb-ae86-180449f264c7-kube-api-access-z6qlf\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.103554 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.103528 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.105268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.105241 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.165899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.165862 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qlf\" (UniqueName: \"kubernetes.io/projected/5eb6bcdf-a951-49cb-ae86-180449f264c7-kube-api-access-z6qlf\") pod \"maas-api-revoke-sub-521c0cda-ssctp\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.451787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.451747 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:41.848289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:41.848252 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp"] Nov 03 19:25:42.737427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:42.737385 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" event={"ID":"5eb6bcdf-a951-49cb-ae86-180449f264c7","Type":"ContainerStarted","Data":"8260466b47756877e5d0dd15c538ca64d8b451351a21da4dcee0c68beb09f2ef"} Nov 03 19:25:42.737645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:42.737443 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" event={"ID":"5eb6bcdf-a951-49cb-ae86-180449f264c7","Type":"ContainerStarted","Data":"6cdf7c61945b98fe2526887b6b33ee89656184e04763474c94ee9ca1d188ec65"} Nov 03 19:25:42.859875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:42.859806 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" podStartSLOduration=2.8597870800000003 podStartE2EDuration="2.85978708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:42.856726112 +0000 UTC m=+2872.339722026" watchObservedRunningTime="2026-11-03 19:25:42.85978708 +0000 UTC m=+2872.342782995" Nov 03 19:25:43.745147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:43.745105 4948 generic.go:356] "Generic (PLEG): container finished" podID="5eb6bcdf-a951-49cb-ae86-180449f264c7" containerID="8260466b47756877e5d0dd15c538ca64d8b451351a21da4dcee0c68beb09f2ef" exitCode=0 Nov 03 19:25:43.745339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:43.745189 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" event={"ID":"5eb6bcdf-a951-49cb-ae86-180449f264c7","Type":"ContainerDied","Data":"8260466b47756877e5d0dd15c538ca64d8b451351a21da4dcee0c68beb09f2ef"} Nov 03 19:25:44.894272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:44.894246 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:45.043567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.042540 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-openshift-ca\") pod \"5eb6bcdf-a951-49cb-ae86-180449f264c7\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " Nov 03 19:25:45.043567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.042603 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-xks-ca\") pod \"5eb6bcdf-a951-49cb-ae86-180449f264c7\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " Nov 03 19:25:45.043567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.042761 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb6bcdf-a951-49cb-ae86-180449f264c7-kube-api-access-z6qlf\" (UniqueName: \"kubernetes.io/projected/5eb6bcdf-a951-49cb-ae86-180449f264c7-kube-api-access-z6qlf\") pod \"5eb6bcdf-a951-49cb-ae86-180449f264c7\" (UID: \"5eb6bcdf-a951-49cb-ae86-180449f264c7\") " Nov 03 19:25:45.043567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.043028 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-openshift-ca" pod "5eb6bcdf-a951-49cb-ae86-180449f264c7" (UID: "5eb6bcdf-a951-49cb-ae86-180449f264c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:45.043567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.043244 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.045385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.045353 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-xks-ca" pod "5eb6bcdf-a951-49cb-ae86-180449f264c7" (UID: "5eb6bcdf-a951-49cb-ae86-180449f264c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:45.045511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.045353 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb6bcdf-a951-49cb-ae86-180449f264c7-kube-api-access-z6qlf" pod "5eb6bcdf-a951-49cb-ae86-180449f264c7" (UID: "5eb6bcdf-a951-49cb-ae86-180449f264c7"). InnerVolumeSpecName "kube-api-access-z6qlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:45.144545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.144513 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb6bcdf-a951-49cb-ae86-180449f264c7-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.144545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.144544 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6qlf\" (UniqueName: \"kubernetes.io/projected/5eb6bcdf-a951-49cb-ae86-180449f264c7-kube-api-access-z6qlf\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.759760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.759728 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" Nov 03 19:25:45.759937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.759754 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp" event={"ID":"5eb6bcdf-a951-49cb-ae86-180449f264c7","Type":"ContainerDied","Data":"6cdf7c61945b98fe2526887b6b33ee89656184e04763474c94ee9ca1d188ec65"} Nov 03 19:25:45.759937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:45.759805 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cdf7c61945b98fe2526887b6b33ee89656184e04763474c94ee9ca1d188ec65" Nov 03 19:25:59.166706 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:59.166669 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59"] Nov 03 19:25:59.167234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:25:59.166898 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" containerID="cri-o://0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b" gracePeriod=30 Nov 03 19:26:00.109155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.109114 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz"] Nov 03 19:26:00.109586 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.109573 4948 cpu_manager.go:414] "containerMap: removing container" podUID="5eb6bcdf-a951-49cb-ae86-180449f264c7" containerName="revoke-keys" Nov 03 19:26:00.109634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.109587 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb6bcdf-a951-49cb-ae86-180449f264c7" containerName="revoke-keys" Nov 03 19:26:00.109634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.109611 4948 cpu_manager.go:414] "containerMap: removing container" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" Nov 03 19:26:00.109634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.109617 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" Nov 03 19:26:00.109750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.109723 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb6bcdf-a951-49cb-ae86-180449f264c7" containerName="revoke-keys" Nov 03 19:26:00.109750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.109740 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="42791c9a-d4ed-4055-b330-f565ef971dc7" containerName="istio-proxy" Nov 03 19:26:00.113424 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.113407 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.167075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.167037 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz"] Nov 03 19:26:00.199466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.199434 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.199629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.199472 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pjl\" (UniqueName: \"kubernetes.io/projected/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-kube-api-access-p9pjl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.199629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.199498 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.214833 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.214805 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:26:00.214958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.214898 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:26:00.300563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.300527 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.300563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.300564 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pjl\" (UniqueName: \"kubernetes.io/projected/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-kube-api-access-p9pjl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.300815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.300594 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.301214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.301181 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.303095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.303065 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.363057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.362992 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pjl\" (UniqueName: \"kubernetes.io/projected/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-kube-api-access-p9pjl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.427149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.427118 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:00.808433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.808111 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz"] Nov 03 19:26:00.810808 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:26:00.810771 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551e41d4_7aa3_4d1c_8deb_b5bb205f6927.slice/crio-8a0dab62fc9612de83ce311c5d80987bcdfee07fad9aa8260d582dd6af922977 WatchSource:0}: Error finding container 8a0dab62fc9612de83ce311c5d80987bcdfee07fad9aa8260d582dd6af922977: Status 404 returned error can't find the container with id 8a0dab62fc9612de83ce311c5d80987bcdfee07fad9aa8260d582dd6af922977 Nov 03 19:26:00.855748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:00.855722 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" event={"ID":"551e41d4-7aa3-4d1c-8deb-b5bb205f6927","Type":"ContainerStarted","Data":"8a0dab62fc9612de83ce311c5d80987bcdfee07fad9aa8260d582dd6af922977"} Nov 03 19:26:01.872356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:01.872267 4948 generic.go:356] "Generic (PLEG): container finished" podID="551e41d4-7aa3-4d1c-8deb-b5bb205f6927" containerID="6a11e94c5685446378a3deda7105d9fb3a6aee1a77966ddb146b8008752f61e1" exitCode=0 Nov 03 19:26:01.872767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:01.872445 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" event={"ID":"551e41d4-7aa3-4d1c-8deb-b5bb205f6927","Type":"ContainerDied","Data":"6a11e94c5685446378a3deda7105d9fb3a6aee1a77966ddb146b8008752f61e1"} Nov 03 19:26:02.795034 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:02.794994 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:26:03.030395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.030364 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:03.132369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.132282 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-kube-api-access-p9pjl\" (UniqueName: \"kubernetes.io/projected/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-kube-api-access-p9pjl\") pod \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " Nov 03 19:26:03.132369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.132366 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-xks-ca\") pod \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " Nov 03 19:26:03.132626 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.132433 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-openshift-ca\") pod \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\" (UID: \"551e41d4-7aa3-4d1c-8deb-b5bb205f6927\") " Nov 03 19:26:03.132886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.132856 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-openshift-ca" pod "551e41d4-7aa3-4d1c-8deb-b5bb205f6927" (UID: "551e41d4-7aa3-4d1c-8deb-b5bb205f6927"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:03.134539 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.134501 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-xks-ca" pod "551e41d4-7aa3-4d1c-8deb-b5bb205f6927" (UID: "551e41d4-7aa3-4d1c-8deb-b5bb205f6927"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:03.134654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.134565 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-kube-api-access-p9pjl" pod "551e41d4-7aa3-4d1c-8deb-b5bb205f6927" (UID: "551e41d4-7aa3-4d1c-8deb-b5bb205f6927"). InnerVolumeSpecName "kube-api-access-p9pjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:03.137787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.137763 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz"] Nov 03 19:26:03.140343 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.140321 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-355e1a-mgrnz"] Nov 03 19:26:03.233228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.233190 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:03.233228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.233223 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:03.233448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.233239 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9pjl\" (UniqueName: \"kubernetes.io/projected/551e41d4-7aa3-4d1c-8deb-b5bb205f6927-kube-api-access-p9pjl\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:03.893813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.893781 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" Nov 03 19:26:03.893998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.893783 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz" event={"ID":"551e41d4-7aa3-4d1c-8deb-b5bb205f6927","Type":"ContainerDied","Data":"8a0dab62fc9612de83ce311c5d80987bcdfee07fad9aa8260d582dd6af922977"} Nov 03 19:26:03.893998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:03.893930 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0dab62fc9612de83ce311c5d80987bcdfee07fad9aa8260d582dd6af922977" Nov 03 19:26:04.417171 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.417142 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:26:04.443946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.443918 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-podinfo\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.443985 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-token\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-token\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444082 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-certs\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444129 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-socket\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444150 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-data\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444221 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-credential-socket\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444271 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istiod-ca-cert\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444321 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-kube-api-access-44mn8\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-kube-api-access-44mn8\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444373 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-envoy\") pod \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\" (UID: \"ac010993-8cde-4557-a9fe-ec54ab8dc11b\") " Nov 03 19:26:04.444546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444391 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-certs" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:04.444546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444375 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-socket" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:04.444546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444444 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-data" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:04.444546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444524 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-credential-socket" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:04.444813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444768 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istiod-ca-cert" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:04.444813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444786 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.444813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444811 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.444926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444826 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.444926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.444840 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.446249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.446226 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-podinfo" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:26:04.446360 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.446334 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-kube-api-access-44mn8" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "kube-api-access-44mn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:04.446583 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.446564 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-token" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:04.446680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.446660 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-envoy" pod "ac010993-8cde-4557-a9fe-ec54ab8dc11b" (UID: "ac010993-8cde-4557-a9fe-ec54ab8dc11b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:04.546114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.546084 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.546114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.546110 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44mn8\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-kube-api-access-44mn8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.546114 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.546120 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.546404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.546132 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.546404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.546141 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac010993-8cde-4557-a9fe-ec54ab8dc11b-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:04.752408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.752335 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f4c2aa-ff06-4229-8427-6235312d96cb" path="/var/lib/kubelet/pods/86f4c2aa-ff06-4229-8427-6235312d96cb/volumes" Nov 03 19:26:04.901351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.901310 4948 generic.go:356] "Generic (PLEG): container finished" podID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerID="0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b" exitCode=0 Nov 03 19:26:04.901502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.901369 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" Nov 03 19:26:04.901502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.901398 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" event={"ID":"ac010993-8cde-4557-a9fe-ec54ab8dc11b","Type":"ContainerDied","Data":"0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b"} Nov 03 19:26:04.901502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.901461 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59" event={"ID":"ac010993-8cde-4557-a9fe-ec54ab8dc11b","Type":"ContainerDied","Data":"3f0c3d52fb843b2d486e713db9b956d71eae6386f87f4f189ea181821599c312"} Nov 03 19:26:04.901502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.901484 4948 scope.go:122] "RemoveContainer" containerID="0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b" Nov 03 19:26:04.911271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.911250 4948 scope.go:122] "RemoveContainer" containerID="17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95" Nov 03 19:26:04.920469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.920452 4948 scope.go:122] "RemoveContainer" containerID="0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b" Nov 03 19:26:04.920781 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:04.920762 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b\": container with ID starting with 0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b not found: ID does not exist" containerID="0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b" Nov 03 19:26:04.920880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.920796 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b"} err="failed to get container status \"0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b\": rpc error: code = NotFound desc = could not find container \"0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b\": container with ID starting with 0d8f0e52cc5fb520eb5584f2b81013674077d85c0c1282b5311c7eb9e008664b not found: ID does not exist" Nov 03 19:26:04.920880 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.920815 4948 scope.go:122] "RemoveContainer" containerID="17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95" Nov 03 19:26:04.921075 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:04.921054 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95\": container with ID starting with 17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95 not found: ID does not exist" containerID="17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95" Nov 03 19:26:04.921138 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:04.921085 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95"} err="failed to get container status \"17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95\": rpc error: code = NotFound desc = could not find container \"17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95\": container with ID starting with 17eb9ebe4edc79ecadcbcfca9dbfd2ce5679db88f9bb452c9b5a3e40f616be95 not found: ID does not exist" Nov 03 19:26:05.073426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:05.073353 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59"] Nov 03 19:26:05.124274 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:05.124242 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770e67-openshift-default-974c475f5-dlf59"] Nov 03 19:26:06.752612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:06.752579 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" path="/var/lib/kubelet/pods/ac010993-8cde-4557-a9fe-ec54ab8dc11b/volumes" Nov 03 19:26:10.187248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:10.187211 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d"] Nov 03 19:26:10.187744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:10.187476 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" containerID="cri-o://2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39" gracePeriod=30 Nov 03 19:26:11.636146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:11.636109 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht"] Nov 03 19:26:11.744768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:11.744737 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl"] Nov 03 19:26:12.113648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:12.113615 4948 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:26:12.113830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:12.113672 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:26:15.457557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.457496 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:26:15.553768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.553732 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-credential-socket\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.553946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.553795 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-certs\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.553946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.553832 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-data\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.553946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.553888 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-podinfo\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.553946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.553914 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-socket\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.554170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.553986 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-kube-api-access-vhdgc\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-kube-api-access-vhdgc\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.554170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.553989 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-credential-socket" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:15.554170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554075 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-envoy\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.554170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554105 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-certs" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:15.554170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554107 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-token\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-token\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.554170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554161 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-socket" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:15.554495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554187 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istiod-ca-cert\") pod \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\" (UID: \"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c\") " Nov 03 19:26:15.554495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554224 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-data" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:15.554605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554537 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.554605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554551 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.554605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554570 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.554605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554583 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.554840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.554597 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istiod-ca-cert" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:15.556392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.556361 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-token" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:15.556392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.556385 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-kube-api-access-vhdgc" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "kube-api-access-vhdgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:15.556578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.556459 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-podinfo" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:26:15.556644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.556578 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-envoy" pod "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" (UID: "7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:15.655440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.655401 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.655440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.655438 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdgc\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-kube-api-access-vhdgc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.655717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.655453 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.655717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.655467 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.655717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.655480 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:15.977765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.977733 4948 generic.go:356] "Generic (PLEG): container finished" podID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerID="2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39" exitCode=0 Nov 03 19:26:15.977765 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.977767 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" Nov 03 19:26:15.978054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.977808 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" event={"ID":"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c","Type":"ContainerDied","Data":"2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39"} Nov 03 19:26:15.978054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.977872 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d" event={"ID":"7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c","Type":"ContainerDied","Data":"8a1badbcf9fe072d85f37ffcc70b6b9c6533a93efd33847d1b242ef20a8af039"} Nov 03 19:26:15.978054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.977902 4948 scope.go:122] "RemoveContainer" containerID="2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39" Nov 03 19:26:15.988900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.988886 4948 scope.go:122] "RemoveContainer" containerID="2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39" Nov 03 19:26:15.989182 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:15.989158 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39\": container with ID starting with 2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39 not found: ID does not exist" containerID="2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39" Nov 03 19:26:15.989287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:15.989188 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39"} err="failed to get container status \"2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39\": rpc error: code = NotFound desc = could not find container \"2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39\": container with ID starting with 2abc5ea1e81be3597cf188a1aae2de51869a6a9d484f12ffd4883aaae2473f39 not found: ID does not exist" Nov 03 19:26:16.148526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.148484 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d"] Nov 03 19:26:16.151525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.151502 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-j6l6d"] Nov 03 19:26:16.636691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.636650 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" podUID="18dd560f-7ce0-46ef-a13a-895c10d116d3" containerName="payload-processing" containerID="cri-o://6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf" gracePeriod=25 Nov 03 19:26:16.745725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.745649 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" podUID="14d087b8-82fd-41c7-b0a7-786552d2b041" containerName="payload-pre-processing" containerID="cri-o://0291c2b359ffe391cfaf31a18d6388fe5d5a4b51b6a843bf1268ebe8523464c4" gracePeriod=25 Nov 03 19:26:16.754743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.754692 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" path="/var/lib/kubelet/pods/7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c/volumes" Nov 03 19:26:16.912878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.912848 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:26:16.969788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.969759 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18dd560f-7ce0-46ef-a13a-895c10d116d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd560f-7ce0-46ef-a13a-895c10d116d3-plugins-config-volume\") pod \"18dd560f-7ce0-46ef-a13a-895c10d116d3\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " Nov 03 19:26:16.969987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.969847 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18dd560f-7ce0-46ef-a13a-895c10d116d3-kube-api-access-xvr66\" (UniqueName: \"kubernetes.io/projected/18dd560f-7ce0-46ef-a13a-895c10d116d3-kube-api-access-xvr66\") pod \"18dd560f-7ce0-46ef-a13a-895c10d116d3\" (UID: \"18dd560f-7ce0-46ef-a13a-895c10d116d3\") " Nov 03 19:26:16.970548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.970195 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18dd560f-7ce0-46ef-a13a-895c10d116d3-plugins-config-volume" pod "18dd560f-7ce0-46ef-a13a-895c10d116d3" (UID: "18dd560f-7ce0-46ef-a13a-895c10d116d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:16.973022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.972983 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18dd560f-7ce0-46ef-a13a-895c10d116d3-kube-api-access-xvr66" pod "18dd560f-7ce0-46ef-a13a-895c10d116d3" (UID: "18dd560f-7ce0-46ef-a13a-895c10d116d3"). InnerVolumeSpecName "kube-api-access-xvr66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:16.986683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.986658 4948 generic.go:356] "Generic (PLEG): container finished" podID="18dd560f-7ce0-46ef-a13a-895c10d116d3" containerID="6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf" exitCode=0 Nov 03 19:26:16.986806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.986734 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" Nov 03 19:26:16.986806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.986754 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" event={"ID":"18dd560f-7ce0-46ef-a13a-895c10d116d3","Type":"ContainerDied","Data":"6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf"} Nov 03 19:26:16.986930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.986821 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht" event={"ID":"18dd560f-7ce0-46ef-a13a-895c10d116d3","Type":"ContainerDied","Data":"e56e7f462a3018c006b90cadab3b6bade1a834d1ad66bc72d7697a7cebead85f"} Nov 03 19:26:16.986930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.986855 4948 scope.go:122] "RemoveContainer" containerID="6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf" Nov 03 19:26:16.989206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.989184 4948 generic.go:356] "Generic (PLEG): container finished" podID="14d087b8-82fd-41c7-b0a7-786552d2b041" containerID="0291c2b359ffe391cfaf31a18d6388fe5d5a4b51b6a843bf1268ebe8523464c4" exitCode=0 Nov 03 19:26:16.989293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:16.989219 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" event={"ID":"14d087b8-82fd-41c7-b0a7-786552d2b041","Type":"ContainerDied","Data":"0291c2b359ffe391cfaf31a18d6388fe5d5a4b51b6a843bf1268ebe8523464c4"} Nov 03 19:26:17.003265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.003246 4948 scope.go:122] "RemoveContainer" containerID="6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf" Nov 03 19:26:17.003803 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:17.003778 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf\": container with ID starting with 6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf not found: ID does not exist" containerID="6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf" Nov 03 19:26:17.003918 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.003814 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf"} err="failed to get container status \"6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf\": rpc error: code = NotFound desc = could not find container \"6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf\": container with ID starting with 6dfccad1243f045a549fa89343e3e48048942d88d71615db69f9fd67520788bf not found: ID does not exist" Nov 03 19:26:17.012877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.012857 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:26:17.071505 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.071465 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d087b8-82fd-41c7-b0a7-786552d2b041-kube-api-access-xfqpr\" (UniqueName: \"kubernetes.io/projected/14d087b8-82fd-41c7-b0a7-786552d2b041-kube-api-access-xfqpr\") pod \"14d087b8-82fd-41c7-b0a7-786552d2b041\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " Nov 03 19:26:17.071691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.071538 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d087b8-82fd-41c7-b0a7-786552d2b041-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d087b8-82fd-41c7-b0a7-786552d2b041-plugins-config-volume\") pod \"14d087b8-82fd-41c7-b0a7-786552d2b041\" (UID: \"14d087b8-82fd-41c7-b0a7-786552d2b041\") " Nov 03 19:26:17.071904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.071883 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd560f-7ce0-46ef-a13a-895c10d116d3-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:17.072048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.071908 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvr66\" (UniqueName: \"kubernetes.io/projected/18dd560f-7ce0-46ef-a13a-895c10d116d3-kube-api-access-xvr66\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:17.072048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.071955 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d087b8-82fd-41c7-b0a7-786552d2b041-plugins-config-volume" pod "14d087b8-82fd-41c7-b0a7-786552d2b041" (UID: "14d087b8-82fd-41c7-b0a7-786552d2b041"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:17.073760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.073732 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d087b8-82fd-41c7-b0a7-786552d2b041-kube-api-access-xfqpr" pod "14d087b8-82fd-41c7-b0a7-786552d2b041" (UID: "14d087b8-82fd-41c7-b0a7-786552d2b041"). InnerVolumeSpecName "kube-api-access-xfqpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:17.173294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.173220 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfqpr\" (UniqueName: \"kubernetes.io/projected/14d087b8-82fd-41c7-b0a7-786552d2b041-kube-api-access-xfqpr\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:17.173294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.173251 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d087b8-82fd-41c7-b0a7-786552d2b041-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:17.268802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.268753 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht"] Nov 03 19:26:17.272610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.272581 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b485d8-657bb4cc85-6p2ht"] Nov 03 19:26:17.999728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.999685 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" Nov 03 19:26:17.999728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.999715 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl" event={"ID":"14d087b8-82fd-41c7-b0a7-786552d2b041","Type":"ContainerDied","Data":"0e74834ef1960128e081b4465f0e0906af4abd762b9947a06c8e6245552bed7f"} Nov 03 19:26:18.000305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:17.999776 4948 scope.go:122] "RemoveContainer" containerID="0291c2b359ffe391cfaf31a18d6388fe5d5a4b51b6a843bf1268ebe8523464c4" Nov 03 19:26:18.167522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:18.167485 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl"] Nov 03 19:26:18.218388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:18.218359 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b485d8-d5fd6c795-n8wxl"] Nov 03 19:26:18.753502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:18.753468 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d087b8-82fd-41c7-b0a7-786552d2b041" path="/var/lib/kubelet/pods/14d087b8-82fd-41c7-b0a7-786552d2b041/volumes" Nov 03 19:26:18.753868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:18.753853 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18dd560f-7ce0-46ef-a13a-895c10d116d3" path="/var/lib/kubelet/pods/18dd560f-7ce0-46ef-a13a-895c10d116d3/volumes" Nov 03 19:26:21.071315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.071265 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm"] Nov 03 19:26:21.071866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.071527 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" podUID="9ecbc954-3a56-490f-9087-9422730083ee" containerName="maas-api" containerID="cri-o://8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f" gracePeriod=30 Nov 03 19:26:21.338661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.338638 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:26:21.414535 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.414506 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls\") pod \"9ecbc954-3a56-490f-9087-9422730083ee\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " Nov 03 19:26:21.414722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.414564 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-metrics-certs\") pod \"9ecbc954-3a56-490f-9087-9422730083ee\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " Nov 03 19:26:21.414722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.414674 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ecbc954-3a56-490f-9087-9422730083ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ecbc954-3a56-490f-9087-9422730083ee-openshift-service-ca\") pod \"9ecbc954-3a56-490f-9087-9422730083ee\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " Nov 03 19:26:21.414815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.414740 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ecbc954-3a56-490f-9087-9422730083ee-kube-api-access-9924w\" (UniqueName: \"kubernetes.io/projected/9ecbc954-3a56-490f-9087-9422730083ee-kube-api-access-9924w\") pod \"9ecbc954-3a56-490f-9087-9422730083ee\" (UID: \"9ecbc954-3a56-490f-9087-9422730083ee\") " Nov 03 19:26:21.415089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.415060 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecbc954-3a56-490f-9087-9422730083ee-openshift-service-ca" pod "9ecbc954-3a56-490f-9087-9422730083ee" (UID: "9ecbc954-3a56-490f-9087-9422730083ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:21.416800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.416773 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-metrics-certs" pod "9ecbc954-3a56-490f-9087-9422730083ee" (UID: "9ecbc954-3a56-490f-9087-9422730083ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:21.416913 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.416840 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecbc954-3a56-490f-9087-9422730083ee-kube-api-access-9924w" pod "9ecbc954-3a56-490f-9087-9422730083ee" (UID: "9ecbc954-3a56-490f-9087-9422730083ee"). InnerVolumeSpecName "kube-api-access-9924w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:21.416968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.416913 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls" pod "9ecbc954-3a56-490f-9087-9422730083ee" (UID: "9ecbc954-3a56-490f-9087-9422730083ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:21.515863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.515829 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9924w\" (UniqueName: \"kubernetes.io/projected/9ecbc954-3a56-490f-9087-9422730083ee-kube-api-access-9924w\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:21.515863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.515859 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:21.515863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.515868 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ecbc954-3a56-490f-9087-9422730083ee-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:21.516102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:21.515877 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ecbc954-3a56-490f-9087-9422730083ee-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:22.029286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.029255 4948 generic.go:356] "Generic (PLEG): container finished" podID="9ecbc954-3a56-490f-9087-9422730083ee" containerID="8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f" exitCode=0 Nov 03 19:26:22.029472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.029300 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" Nov 03 19:26:22.029472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.029340 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" event={"ID":"9ecbc954-3a56-490f-9087-9422730083ee","Type":"ContainerDied","Data":"8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f"} Nov 03 19:26:22.029472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.029392 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm" event={"ID":"9ecbc954-3a56-490f-9087-9422730083ee","Type":"ContainerDied","Data":"cbdd97ae2e204d7ae5d8ccc12b83c492a32d5e821f634423b013d7d4fa10ec13"} Nov 03 19:26:22.029472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.029417 4948 scope.go:122] "RemoveContainer" containerID="8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f" Nov 03 19:26:22.040876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.040860 4948 scope.go:122] "RemoveContainer" containerID="8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f" Nov 03 19:26:22.041132 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:22.041111 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f\": container with ID starting with 8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f not found: ID does not exist" containerID="8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f" Nov 03 19:26:22.041207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.041139 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f"} err="failed to get container status \"8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f\": rpc error: code = NotFound desc = could not find container \"8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f\": container with ID starting with 8872e0ea2263ed1319c92cee2b4254b50d49c5276296f91a24a3cc55e0078e4f not found: ID does not exist" Nov 03 19:26:22.198295 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.198263 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm"] Nov 03 19:26:22.249498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.249468 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b485d8-574fffff86-j9jzm"] Nov 03 19:26:22.752214 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:22.752178 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ecbc954-3a56-490f-9087-9422730083ee" path="/var/lib/kubelet/pods/9ecbc954-3a56-490f-9087-9422730083ee/volumes" Nov 03 19:26:23.107868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:23.107781 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz"] Nov 03 19:26:23.111625 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:23.111599 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b485d8-zqjvz"] Nov 03 19:26:24.752567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:24.752529 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551e41d4-7aa3-4d1c-8deb-b5bb205f6927" path="/var/lib/kubelet/pods/551e41d4-7aa3-4d1c-8deb-b5bb205f6927/volumes" Nov 03 19:26:30.302166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:30.302134 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t"] Nov 03 19:26:30.302868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:30.302365 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" containerID="cri-o://951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173" gracePeriod=30 Nov 03 19:26:31.326879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.326787 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9"] Nov 03 19:26:31.327265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327220 4948 cpu_manager.go:414] "containerMap: removing container" podUID="18dd560f-7ce0-46ef-a13a-895c10d116d3" containerName="payload-processing" Nov 03 19:26:31.327265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327232 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18dd560f-7ce0-46ef-a13a-895c10d116d3" containerName="payload-processing" Nov 03 19:26:31.327265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327248 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" Nov 03 19:26:31.327265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327253 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" Nov 03 19:26:31.327265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327267 4948 cpu_manager.go:414] "containerMap: removing container" podUID="551e41d4-7aa3-4d1c-8deb-b5bb205f6927" containerName="revoke-keys" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327273 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551e41d4-7aa3-4d1c-8deb-b5bb205f6927" containerName="revoke-keys" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327285 4948 cpu_manager.go:414] "containerMap: removing container" podUID="14d087b8-82fd-41c7-b0a7-786552d2b041" containerName="payload-pre-processing" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327290 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d087b8-82fd-41c7-b0a7-786552d2b041" containerName="payload-pre-processing" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327300 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9ecbc954-3a56-490f-9087-9422730083ee" containerName="maas-api" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327306 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ecbc954-3a56-490f-9087-9422730083ee" containerName="maas-api" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327319 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327323 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327334 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327338 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327404 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="551e41d4-7aa3-4d1c-8deb-b5bb205f6927" containerName="revoke-keys" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327413 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d087b8-82fd-41c7-b0a7-786552d2b041" containerName="payload-pre-processing" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327421 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ecbc954-3a56-490f-9087-9422730083ee" containerName="maas-api" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327428 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" Nov 03 19:26:31.327433 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327434 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7138e7af-3d7a-4c4e-9ce5-e26f2a024d0c" containerName="istio-proxy" Nov 03 19:26:31.328000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.327442 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="18dd560f-7ce0-46ef-a13a-895c10d116d3" containerName="payload-processing" Nov 03 19:26:31.332140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.332121 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.386451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.386407 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9"] Nov 03 19:26:31.433435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.433397 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:26:31.433610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.433475 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:26:31.433610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.433538 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:26:31.433610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.433578 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:26:31.513588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.513553 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.513787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.513739 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slb9\" (UniqueName: \"kubernetes.io/projected/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-kube-api-access-9slb9\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.513858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.513804 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.614680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.614600 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slb9\" (UniqueName: \"kubernetes.io/projected/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-kube-api-access-9slb9\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.614680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.614644 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.614930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.614694 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.615422 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.615395 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.617254 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.617230 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.677243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.677208 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slb9\" (UniqueName: \"kubernetes.io/projected/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-kube-api-access-9slb9\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:31.945462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:31.945432 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:32.322858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:32.322827 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9"] Nov 03 19:26:32.324929 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:26:32.324899 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7fcccc_df94_4c1b_a1ab_d8ab719f5aa2.slice/crio-bf502b8b62dd6ac1718c9e25033870071612205f9f3e2237dd4a818aa6fc0adf WatchSource:0}: Error finding container bf502b8b62dd6ac1718c9e25033870071612205f9f3e2237dd4a818aa6fc0adf: Status 404 returned error can't find the container with id bf502b8b62dd6ac1718c9e25033870071612205f9f3e2237dd4a818aa6fc0adf Nov 03 19:26:32.792542 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:32.792500 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:26:33.111448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:33.111407 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" event={"ID":"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2","Type":"ContainerStarted","Data":"3339801d61b9fb360c34387a4b274d5b372b9fd29c9cbd7f1e066e399ae8e741"} Nov 03 19:26:33.111630 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:33.111469 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" event={"ID":"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2","Type":"ContainerStarted","Data":"bf502b8b62dd6ac1718c9e25033870071612205f9f3e2237dd4a818aa6fc0adf"} Nov 03 19:26:33.230653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:33.230603 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" podStartSLOduration=2.230588917 podStartE2EDuration="2.230588917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:33.229367567 +0000 UTC m=+2922.712363481" watchObservedRunningTime="2026-11-03 19:26:33.230588917 +0000 UTC m=+2922.713584830" Nov 03 19:26:34.119584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:34.119552 4948 generic.go:356] "Generic (PLEG): container finished" podID="bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" containerID="3339801d61b9fb360c34387a4b274d5b372b9fd29c9cbd7f1e066e399ae8e741" exitCode=0 Nov 03 19:26:34.120020 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:34.119644 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" event={"ID":"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2","Type":"ContainerDied","Data":"3339801d61b9fb360c34387a4b274d5b372b9fd29c9cbd7f1e066e399ae8e741"} Nov 03 19:26:35.264709 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.264672 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:35.451094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.451064 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-kube-api-access-9slb9\" (UniqueName: \"kubernetes.io/projected/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-kube-api-access-9slb9\") pod \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " Nov 03 19:26:35.451283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.451145 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-xks-ca\") pod \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " Nov 03 19:26:35.451283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.451177 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-openshift-ca\") pod \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\" (UID: \"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2\") " Nov 03 19:26:35.451653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.451623 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-openshift-ca" pod "bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" (UID: "bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:35.453107 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.453083 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-kube-api-access-9slb9" pod "bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" (UID: "bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2"). InnerVolumeSpecName "kube-api-access-9slb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:35.453247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.453232 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-xks-ca" pod "bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" (UID: "bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:35.540276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.540250 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:26:35.552477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.552448 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.552477 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.552473 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.552651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.552483 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9slb9\" (UniqueName: \"kubernetes.io/projected/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2-kube-api-access-9slb9\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.652907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.652872 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-credential-socket\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.652931 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-istio-token\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-istio-token\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.652976 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-envoy\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653017 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-certs\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653063 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce38743-6104-4500-8cf0-0806409dce6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce38743-6104-4500-8cf0-0806409dce6e-istiod-ca-cert\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653148 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-kube-api-access-5mcmx\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-kube-api-access-5mcmx\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653188 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-data\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653219 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-credential-socket" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:35.653284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653265 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-socket\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653327 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce38743-6104-4500-8cf0-0806409dce6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce38743-6104-4500-8cf0-0806409dce6e-istio-podinfo\") pod \"dce38743-6104-4500-8cf0-0806409dce6e\" (UID: \"dce38743-6104-4500-8cf0-0806409dce6e\") " Nov 03 19:26:35.653510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653446 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-data" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:35.653745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653646 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-socket" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:35.653745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653688 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-certs" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:35.653745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.653735 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce38743-6104-4500-8cf0-0806409dce6e-istiod-ca-cert" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:35.654116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.654094 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.654232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.654122 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.654232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.654140 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.655554 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.655528 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-envoy" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:35.655554 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.655544 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce38743-6104-4500-8cf0-0806409dce6e-istio-podinfo" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:26:35.655767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.655747 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-kube-api-access-5mcmx" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "kube-api-access-5mcmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:35.655767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.655758 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-istio-token" pod "dce38743-6104-4500-8cf0-0806409dce6e" (UID: "dce38743-6104-4500-8cf0-0806409dce6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:35.753634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.753599 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl"] Nov 03 19:26:35.755577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.755548 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.755577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.755574 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce38743-6104-4500-8cf0-0806409dce6e-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.755789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.755584 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce38743-6104-4500-8cf0-0806409dce6e-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.755789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.755593 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcmx\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-kube-api-access-5mcmx\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.755789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.755602 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce38743-6104-4500-8cf0-0806409dce6e-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.755789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.755610 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce38743-6104-4500-8cf0-0806409dce6e-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:35.781999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:35.781968 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn"] Nov 03 19:26:36.134875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.134787 4948 generic.go:356] "Generic (PLEG): container finished" podID="dce38743-6104-4500-8cf0-0806409dce6e" containerID="951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173" exitCode=0 Nov 03 19:26:36.134875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.134864 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" Nov 03 19:26:36.135092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.134874 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" event={"ID":"dce38743-6104-4500-8cf0-0806409dce6e","Type":"ContainerDied","Data":"951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173"} Nov 03 19:26:36.135092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.134969 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t" event={"ID":"dce38743-6104-4500-8cf0-0806409dce6e","Type":"ContainerDied","Data":"08e7eed38c8016b6992006cab3e46e2c1d67c01517894ffbefd3c5839b8ec66c"} Nov 03 19:26:36.135092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.135001 4948 scope.go:122] "RemoveContainer" containerID="951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173" Nov 03 19:26:36.136669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.136646 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" event={"ID":"bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2","Type":"ContainerDied","Data":"bf502b8b62dd6ac1718c9e25033870071612205f9f3e2237dd4a818aa6fc0adf"} Nov 03 19:26:36.136852 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.136833 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf502b8b62dd6ac1718c9e25033870071612205f9f3e2237dd4a818aa6fc0adf" Nov 03 19:26:36.136852 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.136676 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9" Nov 03 19:26:36.146912 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.146844 4948 scope.go:122] "RemoveContainer" containerID="6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87" Nov 03 19:26:36.156369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.156344 4948 scope.go:122] "RemoveContainer" containerID="951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173" Nov 03 19:26:36.156622 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:36.156601 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173\": container with ID starting with 951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173 not found: ID does not exist" containerID="951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173" Nov 03 19:26:36.156788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.156630 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173"} err="failed to get container status \"951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173\": rpc error: code = NotFound desc = could not find container \"951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173\": container with ID starting with 951680e78462e05e31a2487dc5d336dac49e9d3a79daacbd7ed97f03f65b8173 not found: ID does not exist" Nov 03 19:26:36.156788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.156648 4948 scope.go:122] "RemoveContainer" containerID="6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87" Nov 03 19:26:36.156917 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:36.156880 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87\": container with ID starting with 6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87 not found: ID does not exist" containerID="6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87" Nov 03 19:26:36.156917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.156902 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87"} err="failed to get container status \"6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87\": rpc error: code = NotFound desc = could not find container \"6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87\": container with ID starting with 6e2a94e5d32e5861b110e7ed1eea5bf4f8c18aa4cab1286d0608eec8e5de3f87 not found: ID does not exist" Nov 03 19:26:36.358342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.358304 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t"] Nov 03 19:26:36.364336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.364306 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b485d8-openshift-default-788458b655-bff6t"] Nov 03 19:26:36.752229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:36.752189 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce38743-6104-4500-8cf0-0806409dce6e" path="/var/lib/kubelet/pods/dce38743-6104-4500-8cf0-0806409dce6e/volumes" Nov 03 19:26:42.959167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:42.959133 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z"] Nov 03 19:26:47.959458 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:47.959384 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" podUID="2868fe39-a729-4b21-a27f-82c40f95de32" containerName="payload-pre-processing" containerID="cri-o://4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a" gracePeriod=25 Nov 03 19:26:48.213292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.213222 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 19:26:48.217789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.217754 4948 generic.go:356] "Generic (PLEG): container finished" podID="2868fe39-a729-4b21-a27f-82c40f95de32" containerID="4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a" exitCode=0 Nov 03 19:26:48.217941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.217821 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" Nov 03 19:26:48.217941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.217908 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" event={"ID":"2868fe39-a729-4b21-a27f-82c40f95de32","Type":"ContainerDied","Data":"4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a"} Nov 03 19:26:48.218091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.217989 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z" event={"ID":"2868fe39-a729-4b21-a27f-82c40f95de32","Type":"ContainerDied","Data":"277bf53dd637884e6a413246e239bfe0cc595361b099748036c8b250461f114e"} Nov 03 19:26:48.218091 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.218019 4948 scope.go:122] "RemoveContainer" containerID="4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a" Nov 03 19:26:48.227692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.227670 4948 scope.go:122] "RemoveContainer" containerID="4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a" Nov 03 19:26:48.228183 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:48.228155 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a\": container with ID starting with 4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a not found: ID does not exist" containerID="4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a" Nov 03 19:26:48.228276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.228199 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a"} err="failed to get container status \"4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a\": rpc error: code = NotFound desc = could not find container \"4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a\": container with ID starting with 4fc84c560106439851eea3b0e7207cb6f5fd7afc5dc7abbed9d5b1239999ec6a not found: ID does not exist" Nov 03 19:26:48.290167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.290132 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2868fe39-a729-4b21-a27f-82c40f95de32-kube-api-access-v65k8\" (UniqueName: \"kubernetes.io/projected/2868fe39-a729-4b21-a27f-82c40f95de32-kube-api-access-v65k8\") pod \"2868fe39-a729-4b21-a27f-82c40f95de32\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " Nov 03 19:26:48.290359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.290251 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2868fe39-a729-4b21-a27f-82c40f95de32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2868fe39-a729-4b21-a27f-82c40f95de32-plugins-config-volume\") pod \"2868fe39-a729-4b21-a27f-82c40f95de32\" (UID: \"2868fe39-a729-4b21-a27f-82c40f95de32\") " Nov 03 19:26:48.290656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.290629 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2868fe39-a729-4b21-a27f-82c40f95de32-plugins-config-volume" pod "2868fe39-a729-4b21-a27f-82c40f95de32" (UID: "2868fe39-a729-4b21-a27f-82c40f95de32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:48.292363 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.292336 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2868fe39-a729-4b21-a27f-82c40f95de32-kube-api-access-v65k8" pod "2868fe39-a729-4b21-a27f-82c40f95de32" (UID: "2868fe39-a729-4b21-a27f-82c40f95de32"). InnerVolumeSpecName "kube-api-access-v65k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:48.391322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.391280 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v65k8\" (UniqueName: \"kubernetes.io/projected/2868fe39-a729-4b21-a27f-82c40f95de32-kube-api-access-v65k8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:48.391322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.391313 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2868fe39-a729-4b21-a27f-82c40f95de32-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:48.644829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.644787 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z"] Nov 03 19:26:48.649362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.649336 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-32a59b-5b54fcd66c-2nl8z"] Nov 03 19:26:48.753370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:48.753335 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2868fe39-a729-4b21-a27f-82c40f95de32" path="/var/lib/kubelet/pods/2868fe39-a729-4b21-a27f-82c40f95de32/volumes" Nov 03 19:26:50.364331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364298 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c"] Nov 03 19:26:50.364799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364784 4948 cpu_manager.go:414] "containerMap: removing container" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" Nov 03 19:26:50.364872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364801 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" Nov 03 19:26:50.364872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364812 4948 cpu_manager.go:414] "containerMap: removing container" podUID="bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" containerName="revoke-keys" Nov 03 19:26:50.364872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364818 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" containerName="revoke-keys" Nov 03 19:26:50.364872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364866 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2868fe39-a729-4b21-a27f-82c40f95de32" containerName="payload-pre-processing" Nov 03 19:26:50.364872 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364871 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2868fe39-a729-4b21-a27f-82c40f95de32" containerName="payload-pre-processing" Nov 03 19:26:50.365031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364938 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" Nov 03 19:26:50.365031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364945 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2868fe39-a729-4b21-a27f-82c40f95de32" containerName="payload-pre-processing" Nov 03 19:26:50.365031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364956 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac010993-8cde-4557-a9fe-ec54ab8dc11b" containerName="istio-proxy" Nov 03 19:26:50.365031 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.364961 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" containerName="revoke-keys" Nov 03 19:26:50.368122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.368102 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.419888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.419858 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp"] Nov 03 19:26:50.420440 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.420420 4948 cpu_manager.go:414] "containerMap: removing container" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" Nov 03 19:26:50.420552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.420442 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" Nov 03 19:26:50.420613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.420560 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce38743-6104-4500-8cf0-0806409dce6e" containerName="istio-proxy" Nov 03 19:26:50.423776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.423754 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.423954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.423928 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c"] Nov 03 19:26:50.467571 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.467533 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:26:50.470591 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.470561 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:26:50.470771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.470601 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:26:50.470771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.470601 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:26:50.476933 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.476907 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp"] Nov 03 19:26:50.509169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.509138 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.509357 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.509200 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.509357 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.509312 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.509455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.509389 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lbg\" (UniqueName: \"kubernetes.io/projected/d6b03e40-65b9-4472-9d70-d03451fed8ee-kube-api-access-z4lbg\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.509455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.509430 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.509532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.509510 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2shp\" (UniqueName: \"kubernetes.io/projected/1d7c4b28-314b-428b-b21a-34b890e5fa31-kube-api-access-k2shp\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.610523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.610479 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.610747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.610558 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lbg\" (UniqueName: \"kubernetes.io/projected/d6b03e40-65b9-4472-9d70-d03451fed8ee-kube-api-access-z4lbg\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.610747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.610592 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.610747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.610627 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2shp\" (UniqueName: \"kubernetes.io/projected/1d7c4b28-314b-428b-b21a-34b890e5fa31-kube-api-access-k2shp\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.610747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.610672 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.611021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.610762 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.611423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.611398 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.611523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.611454 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.613045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.613023 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.613144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.613103 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.675248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.675172 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lbg\" (UniqueName: \"kubernetes.io/projected/d6b03e40-65b9-4472-9d70-d03451fed8ee-kube-api-access-z4lbg\") pod \"maas-api-revoke-sub-cde47b13-qndlp\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.675248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.675173 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2shp\" (UniqueName: \"kubernetes.io/projected/1d7c4b28-314b-428b-b21a-34b890e5fa31-kube-api-access-k2shp\") pod \"maas-api-revoke-sub-3a7e66ce-77z7c\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.680876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.680842 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:50.751737 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.751692 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:50.774930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.774849 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" containerID="cri-o://9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5" gracePeriod=45 Nov 03 19:26:50.803203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:50.803165 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" containerID="cri-o://67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81" gracePeriod=45 Nov 03 19:26:51.043327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.043301 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:26:51.063650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.063388 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c"] Nov 03 19:26:51.066764 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:26:51.066694 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7c4b28_314b_428b_b21a_34b890e5fa31.slice/crio-faac547bb5239da6eaca44aa57564a42df9f50d820b38bd9140d0d7ee9a7ddfa WatchSource:0}: Error finding container faac547bb5239da6eaca44aa57564a42df9f50d820b38bd9140d0d7ee9a7ddfa: Status 404 returned error can't find the container with id faac547bb5239da6eaca44aa57564a42df9f50d820b38bd9140d0d7ee9a7ddfa Nov 03 19:26:51.076317 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.076293 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:26:51.116942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.116911 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-model-cache\") pod \"dbf27b75-38f1-4318-b4f3-4101c02865f5\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " Nov 03 19:26:51.117155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117023 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf27b75-38f1-4318-b4f3-4101c02865f5-kube-api-access-rt8wc\" (UniqueName: \"kubernetes.io/projected/dbf27b75-38f1-4318-b4f3-4101c02865f5-kube-api-access-rt8wc\") pod \"dbf27b75-38f1-4318-b4f3-4101c02865f5\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " Nov 03 19:26:51.117155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117054 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-home\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-home\") pod \"dbf27b75-38f1-4318-b4f3-4101c02865f5\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " Nov 03 19:26:51.117155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117085 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf27b75-38f1-4318-b4f3-4101c02865f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf27b75-38f1-4318-b4f3-4101c02865f5-tls-certs\") pod \"dbf27b75-38f1-4318-b4f3-4101c02865f5\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " Nov 03 19:26:51.117155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117129 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-dshm\") pod \"dbf27b75-38f1-4318-b4f3-4101c02865f5\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " Nov 03 19:26:51.117453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117200 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-tmp-dir\") pod \"dbf27b75-38f1-4318-b4f3-4101c02865f5\" (UID: \"dbf27b75-38f1-4318-b4f3-4101c02865f5\") " Nov 03 19:26:51.117453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117252 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-model-cache" pod "dbf27b75-38f1-4318-b4f3-4101c02865f5" (UID: "dbf27b75-38f1-4318-b4f3-4101c02865f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.117453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117369 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-home" pod "dbf27b75-38f1-4318-b4f3-4101c02865f5" (UID: "dbf27b75-38f1-4318-b4f3-4101c02865f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.117622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117519 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-tmp-dir" pod "dbf27b75-38f1-4318-b4f3-4101c02865f5" (UID: "dbf27b75-38f1-4318-b4f3-4101c02865f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.117622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117607 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.117763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.117629 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.119407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.119380 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-dshm" pod "dbf27b75-38f1-4318-b4f3-4101c02865f5" (UID: "dbf27b75-38f1-4318-b4f3-4101c02865f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.119527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.119440 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf27b75-38f1-4318-b4f3-4101c02865f5-kube-api-access-rt8wc" pod "dbf27b75-38f1-4318-b4f3-4101c02865f5" (UID: "dbf27b75-38f1-4318-b4f3-4101c02865f5"). InnerVolumeSpecName "kube-api-access-rt8wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:51.119527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.119463 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf27b75-38f1-4318-b4f3-4101c02865f5-tls-certs" pod "dbf27b75-38f1-4318-b4f3-4101c02865f5" (UID: "dbf27b75-38f1-4318-b4f3-4101c02865f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:51.173376 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.173347 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp"] Nov 03 19:26:51.218196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218111 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-model-cache\") pod \"90533a1d-b6cf-4600-886b-6429d04c9e2c\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " Nov 03 19:26:51.218377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218209 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-tmp-dir\") pod \"90533a1d-b6cf-4600-886b-6429d04c9e2c\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " Nov 03 19:26:51.218377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218277 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90533a1d-b6cf-4600-886b-6429d04c9e2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/90533a1d-b6cf-4600-886b-6429d04c9e2c-tls-certs\") pod \"90533a1d-b6cf-4600-886b-6429d04c9e2c\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " Nov 03 19:26:51.218377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218322 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90533a1d-b6cf-4600-886b-6429d04c9e2c-kube-api-access-q9vdg\" (UniqueName: \"kubernetes.io/projected/90533a1d-b6cf-4600-886b-6429d04c9e2c-kube-api-access-q9vdg\") pod \"90533a1d-b6cf-4600-886b-6429d04c9e2c\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " Nov 03 19:26:51.218590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218387 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-home\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-home\") pod \"90533a1d-b6cf-4600-886b-6429d04c9e2c\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " Nov 03 19:26:51.218590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218443 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-model-cache" pod "90533a1d-b6cf-4600-886b-6429d04c9e2c" (UID: "90533a1d-b6cf-4600-886b-6429d04c9e2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.218590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218530 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-tmp-dir" pod "90533a1d-b6cf-4600-886b-6429d04c9e2c" (UID: "90533a1d-b6cf-4600-886b-6429d04c9e2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.218806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218773 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-home" pod "90533a1d-b6cf-4600-886b-6429d04c9e2c" (UID: "90533a1d-b6cf-4600-886b-6429d04c9e2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.218957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.218929 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-dshm\") pod \"90533a1d-b6cf-4600-886b-6429d04c9e2c\" (UID: \"90533a1d-b6cf-4600-886b-6429d04c9e2c\") " Nov 03 19:26:51.219373 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.219347 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf27b75-38f1-4318-b4f3-4101c02865f5-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.219484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.219380 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.219484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.219397 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.219484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.219414 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.219484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.219430 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf27b75-38f1-4318-b4f3-4101c02865f5-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.219484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.219446 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.219484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.219467 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt8wc\" (UniqueName: \"kubernetes.io/projected/dbf27b75-38f1-4318-b4f3-4101c02865f5-kube-api-access-rt8wc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.220550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.220521 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90533a1d-b6cf-4600-886b-6429d04c9e2c-tls-certs" pod "90533a1d-b6cf-4600-886b-6429d04c9e2c" (UID: "90533a1d-b6cf-4600-886b-6429d04c9e2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:51.220826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.220800 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90533a1d-b6cf-4600-886b-6429d04c9e2c-kube-api-access-q9vdg" pod "90533a1d-b6cf-4600-886b-6429d04c9e2c" (UID: "90533a1d-b6cf-4600-886b-6429d04c9e2c"). InnerVolumeSpecName "kube-api-access-q9vdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:51.221688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.221669 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-dshm" pod "90533a1d-b6cf-4600-886b-6429d04c9e2c" (UID: "90533a1d-b6cf-4600-886b-6429d04c9e2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.240667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.240628 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" event={"ID":"d6b03e40-65b9-4472-9d70-d03451fed8ee","Type":"ContainerStarted","Data":"3fa187b77d4517eec08192236f6693f26664d354c22de56770871a49be38c25e"} Nov 03 19:26:51.242175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.242147 4948 generic.go:356] "Generic (PLEG): container finished" podID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerID="9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5" exitCode=0 Nov 03 19:26:51.242319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.242218 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" Nov 03 19:26:51.242319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.242231 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" event={"ID":"dbf27b75-38f1-4318-b4f3-4101c02865f5","Type":"ContainerDied","Data":"9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5"} Nov 03 19:26:51.242319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.242273 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl" event={"ID":"dbf27b75-38f1-4318-b4f3-4101c02865f5","Type":"ContainerDied","Data":"c5ce6e4f1e08abb6c389bdec751b946c8e7adfb49eede83b4e98cb690a28dbea"} Nov 03 19:26:51.242319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.242290 4948 scope.go:122] "RemoveContainer" containerID="9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5" Nov 03 19:26:51.243958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.243928 4948 generic.go:356] "Generic (PLEG): container finished" podID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerID="67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81" exitCode=0 Nov 03 19:26:51.244069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.243990 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" Nov 03 19:26:51.244069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.244002 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" event={"ID":"90533a1d-b6cf-4600-886b-6429d04c9e2c","Type":"ContainerDied","Data":"67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81"} Nov 03 19:26:51.244069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.244036 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" event={"ID":"90533a1d-b6cf-4600-886b-6429d04c9e2c","Type":"ContainerDied","Data":"a99dd06029912fb69da421b99eec2da1fec463b790e947ebfb28701942553b6e"} Nov 03 19:26:51.245647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.245581 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" event={"ID":"1d7c4b28-314b-428b-b21a-34b890e5fa31","Type":"ContainerStarted","Data":"faac547bb5239da6eaca44aa57564a42df9f50d820b38bd9140d0d7ee9a7ddfa"} Nov 03 19:26:51.255127 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.255110 4948 scope.go:122] "RemoveContainer" containerID="9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5" Nov 03 19:26:51.255464 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:51.255442 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5\": container with ID starting with 9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5 not found: ID does not exist" containerID="9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5" Nov 03 19:26:51.255555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.255479 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5"} err="failed to get container status \"9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5\": rpc error: code = NotFound desc = could not find container \"9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5\": container with ID starting with 9232d67339d27864134a28c22fe2f245a2cd7652b5a3f3387bc64f9481c47fc5 not found: ID does not exist" Nov 03 19:26:51.255555 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.255506 4948 scope.go:122] "RemoveContainer" containerID="67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81" Nov 03 19:26:51.272055 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.272035 4948 scope.go:122] "RemoveContainer" containerID="67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81" Nov 03 19:26:51.272352 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:26:51.272331 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81\": container with ID starting with 67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81 not found: ID does not exist" containerID="67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81" Nov 03 19:26:51.272426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.272362 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81"} err="failed to get container status \"67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81\": rpc error: code = NotFound desc = could not find container \"67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81\": container with ID starting with 67647b6d38606f263d16808bbdc2f141d2349ef2d5a15490f45530766129ae81 not found: ID does not exist" Nov 03 19:26:51.320595 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.320555 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90533a1d-b6cf-4600-886b-6429d04c9e2c-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.320595 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.320588 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vdg\" (UniqueName: \"kubernetes.io/projected/90533a1d-b6cf-4600-886b-6429d04c9e2c-kube-api-access-q9vdg\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.320821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.320604 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90533a1d-b6cf-4600-886b-6429d04c9e2c-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.396538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.396445 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl"] Nov 03 19:26:51.399742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.399714 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-2-w2-32a59b-kserve-68dcfc9968-5pbfl"] Nov 03 19:26:51.510365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.510277 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn"] Nov 03 19:26:51.514670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:51.514643 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn"] Nov 03 19:26:52.042394 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:52.042348 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/simulator-w2-32a59b-kserve-6888947cb9-5k9rn" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:26:52.264313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:52.264269 4948 generic.go:356] "Generic (PLEG): container finished" podID="1d7c4b28-314b-428b-b21a-34b890e5fa31" containerID="6551d71ece4c55962b1bed089d991115c0451a98b61461afdbfe8e5b4ca5d5f1" exitCode=0 Nov 03 19:26:52.264530 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:52.264348 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" event={"ID":"1d7c4b28-314b-428b-b21a-34b890e5fa31","Type":"ContainerDied","Data":"6551d71ece4c55962b1bed089d991115c0451a98b61461afdbfe8e5b4ca5d5f1"} Nov 03 19:26:52.266182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:52.266153 4948 generic.go:356] "Generic (PLEG): container finished" podID="d6b03e40-65b9-4472-9d70-d03451fed8ee" containerID="52de5a73d36cacdc9a8a2e1244cd18e4fe8ef72660526e2be9b5f4683f2f3a14" exitCode=0 Nov 03 19:26:52.266270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:52.266218 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" event={"ID":"d6b03e40-65b9-4472-9d70-d03451fed8ee","Type":"ContainerDied","Data":"52de5a73d36cacdc9a8a2e1244cd18e4fe8ef72660526e2be9b5f4683f2f3a14"} Nov 03 19:26:52.751922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:52.751885 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" path="/var/lib/kubelet/pods/90533a1d-b6cf-4600-886b-6429d04c9e2c/volumes" Nov 03 19:26:52.752324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:52.752304 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" path="/var/lib/kubelet/pods/dbf27b75-38f1-4318-b4f3-4101c02865f5/volumes" Nov 03 19:26:53.425326 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.425306 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:53.461890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.461853 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:53.544615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.544532 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-openshift-ca\") pod \"1d7c4b28-314b-428b-b21a-34b890e5fa31\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " Nov 03 19:26:53.544842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.544631 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-xks-ca\") pod \"d6b03e40-65b9-4472-9d70-d03451fed8ee\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " Nov 03 19:26:53.544842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.544669 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7c4b28-314b-428b-b21a-34b890e5fa31-kube-api-access-k2shp\" (UniqueName: \"kubernetes.io/projected/1d7c4b28-314b-428b-b21a-34b890e5fa31-kube-api-access-k2shp\") pod \"1d7c4b28-314b-428b-b21a-34b890e5fa31\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " Nov 03 19:26:53.544842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.544724 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-xks-ca\") pod \"1d7c4b28-314b-428b-b21a-34b890e5fa31\" (UID: \"1d7c4b28-314b-428b-b21a-34b890e5fa31\") " Nov 03 19:26:53.544842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.544764 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-openshift-ca\") pod \"d6b03e40-65b9-4472-9d70-d03451fed8ee\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " Nov 03 19:26:53.544842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.544790 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b03e40-65b9-4472-9d70-d03451fed8ee-kube-api-access-z4lbg\" (UniqueName: \"kubernetes.io/projected/d6b03e40-65b9-4472-9d70-d03451fed8ee-kube-api-access-z4lbg\") pod \"d6b03e40-65b9-4472-9d70-d03451fed8ee\" (UID: \"d6b03e40-65b9-4472-9d70-d03451fed8ee\") " Nov 03 19:26:53.545148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.544963 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-openshift-ca" pod "1d7c4b28-314b-428b-b21a-34b890e5fa31" (UID: "1d7c4b28-314b-428b-b21a-34b890e5fa31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:53.545148 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.545072 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:53.545355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.545321 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-openshift-ca" pod "d6b03e40-65b9-4472-9d70-d03451fed8ee" (UID: "d6b03e40-65b9-4472-9d70-d03451fed8ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:53.547339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.547313 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b03e40-65b9-4472-9d70-d03451fed8ee-kube-api-access-z4lbg" pod "d6b03e40-65b9-4472-9d70-d03451fed8ee" (UID: "d6b03e40-65b9-4472-9d70-d03451fed8ee"). InnerVolumeSpecName "kube-api-access-z4lbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:53.547458 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.547385 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-xks-ca" pod "1d7c4b28-314b-428b-b21a-34b890e5fa31" (UID: "1d7c4b28-314b-428b-b21a-34b890e5fa31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:53.547610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.547583 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7c4b28-314b-428b-b21a-34b890e5fa31-kube-api-access-k2shp" pod "1d7c4b28-314b-428b-b21a-34b890e5fa31" (UID: "1d7c4b28-314b-428b-b21a-34b890e5fa31"). InnerVolumeSpecName "kube-api-access-k2shp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:53.547800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.547774 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-xks-ca" pod "d6b03e40-65b9-4472-9d70-d03451fed8ee" (UID: "d6b03e40-65b9-4472-9d70-d03451fed8ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:53.645952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.645911 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:53.645952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.645947 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2shp\" (UniqueName: \"kubernetes.io/projected/1d7c4b28-314b-428b-b21a-34b890e5fa31-kube-api-access-k2shp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:53.645952 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.645958 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7c4b28-314b-428b-b21a-34b890e5fa31-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:53.646212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.645967 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b03e40-65b9-4472-9d70-d03451fed8ee-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:53.646212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:53.645976 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4lbg\" (UniqueName: \"kubernetes.io/projected/d6b03e40-65b9-4472-9d70-d03451fed8ee-kube-api-access-z4lbg\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:54.282766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:54.282732 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" Nov 03 19:26:54.282766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:54.282748 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp" event={"ID":"d6b03e40-65b9-4472-9d70-d03451fed8ee","Type":"ContainerDied","Data":"3fa187b77d4517eec08192236f6693f26664d354c22de56770871a49be38c25e"} Nov 03 19:26:54.283253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:54.282803 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fa187b77d4517eec08192236f6693f26664d354c22de56770871a49be38c25e" Nov 03 19:26:54.284305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:54.284278 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" event={"ID":"1d7c4b28-314b-428b-b21a-34b890e5fa31","Type":"ContainerDied","Data":"faac547bb5239da6eaca44aa57564a42df9f50d820b38bd9140d0d7ee9a7ddfa"} Nov 03 19:26:54.284451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:54.284314 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c" Nov 03 19:26:54.284451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:26:54.284316 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faac547bb5239da6eaca44aa57564a42df9f50d820b38bd9140d0d7ee9a7ddfa" Nov 03 19:27:02.793427 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:27:02.793388 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:27:07.735191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:07.735152 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9"] Nov 03 19:27:07.785631 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:07.785599 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a59b-mzqf9"] Nov 03 19:27:08.752219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:08.752179 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2" path="/var/lib/kubelet/pods/bd7fcccc-df94-4c1b-a1ab-d8ab719f5aa2/volumes" Nov 03 19:27:20.611948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.611907 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68f76587d6-m4k75"] Nov 03 19:27:20.612417 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.612196 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68f76587d6-m4k75" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerName="manager" containerID="cri-o://1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2" gracePeriod=30 Nov 03 19:27:20.873198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.873135 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 19:27:20.898970 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.898931 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-kube-api-access-xgf9h\" (UniqueName: \"kubernetes.io/projected/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-kube-api-access-xgf9h\") pod \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " Nov 03 19:27:20.899164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.899001 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-webhook-cert\") pod \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " Nov 03 19:27:20.899164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.899066 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-metrics-certs\") pod \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\" (UID: \"9aaae9b4-e7dc-4cdb-bb80-9a4639354338\") " Nov 03 19:27:20.901249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.901214 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-kube-api-access-xgf9h" pod "9aaae9b4-e7dc-4cdb-bb80-9a4639354338" (UID: "9aaae9b4-e7dc-4cdb-bb80-9a4639354338"). InnerVolumeSpecName "kube-api-access-xgf9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:20.901249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.901227 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-webhook-cert" pod "9aaae9b4-e7dc-4cdb-bb80-9a4639354338" (UID: "9aaae9b4-e7dc-4cdb-bb80-9a4639354338"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:20.901410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:20.901238 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-metrics-certs" pod "9aaae9b4-e7dc-4cdb-bb80-9a4639354338" (UID: "9aaae9b4-e7dc-4cdb-bb80-9a4639354338"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:21.000537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.000496 4948 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-webhook-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:21.000537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.000537 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:21.000849 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.000554 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgf9h\" (UniqueName: \"kubernetes.io/projected/9aaae9b4-e7dc-4cdb-bb80-9a4639354338-kube-api-access-xgf9h\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:21.452903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.452866 4948 generic.go:356] "Generic (PLEG): container finished" podID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerID="1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2" exitCode=0 Nov 03 19:27:21.453100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.452948 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-m4k75" event={"ID":"9aaae9b4-e7dc-4cdb-bb80-9a4639354338","Type":"ContainerDied","Data":"1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2"} Nov 03 19:27:21.453100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.452998 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-m4k75" Nov 03 19:27:21.453100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.453029 4948 scope.go:122] "RemoveContainer" containerID="1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2" Nov 03 19:27:21.453280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.453008 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-m4k75" event={"ID":"9aaae9b4-e7dc-4cdb-bb80-9a4639354338","Type":"ContainerDied","Data":"815bfeb0a5ec7ba5785682571e561b441945ce8e585230ac1b1bb91bfb342bb3"} Nov 03 19:27:21.463647 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.463626 4948 scope.go:122] "RemoveContainer" containerID="1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7" Nov 03 19:27:21.472726 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.472684 4948 scope.go:122] "RemoveContainer" containerID="1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2" Nov 03 19:27:21.473060 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:27:21.473041 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2\": container with ID starting with 1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2 not found: ID does not exist" containerID="1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2" Nov 03 19:27:21.473144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.473075 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2"} err="failed to get container status \"1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2\": rpc error: code = NotFound desc = could not find container \"1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2\": container with ID starting with 1bdc5dace93c2a2b18bd1404ad5c8706c43e4054311a264691eaa6346f4674c2 not found: ID does not exist" Nov 03 19:27:21.473144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.473093 4948 scope.go:122] "RemoveContainer" containerID="1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7" Nov 03 19:27:21.473358 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:27:21.473339 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7\": container with ID starting with 1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7 not found: ID does not exist" containerID="1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7" Nov 03 19:27:21.473405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.473365 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7"} err="failed to get container status \"1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7\": rpc error: code = NotFound desc = could not find container \"1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7\": container with ID starting with 1f4779bc341ccdeb7276d9c8fd95adee629bdb66fd4670d3ed7ecc0fb2f233c7 not found: ID does not exist" Nov 03 19:27:21.621861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.621820 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68f76587d6-m4k75"] Nov 03 19:27:21.672890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:21.672854 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68f76587d6-m4k75"] Nov 03 19:27:22.510995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.510952 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68f76587d6-bzh4v"] Nov 03 19:27:22.511389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511373 4948 cpu_manager.go:414] "containerMap: removing container" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" Nov 03 19:27:22.511389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511387 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511398 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerName="manager" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511404 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerName="manager" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511414 4948 cpu_manager.go:414] "containerMap: removing container" podUID="d6b03e40-65b9-4472-9d70-d03451fed8ee" containerName="revoke-keys" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511419 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b03e40-65b9-4472-9d70-d03451fed8ee" containerName="revoke-keys" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511431 4948 cpu_manager.go:414] "containerMap: removing container" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511435 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511442 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerName="manager" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511447 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerName="manager" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511463 4948 cpu_manager.go:414] "containerMap: removing container" podUID="1d7c4b28-314b-428b-b21a-34b890e5fa31" containerName="revoke-keys" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511468 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7c4b28-314b-428b-b21a-34b890e5fa31" containerName="revoke-keys" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511524 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7c4b28-314b-428b-b21a-34b890e5fa31" containerName="revoke-keys" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511535 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="90533a1d-b6cf-4600-886b-6429d04c9e2c" containerName="main" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511540 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerName="manager" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511546 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b03e40-65b9-4472-9d70-d03451fed8ee" containerName="revoke-keys" Nov 03 19:27:22.511545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.511552 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf27b75-38f1-4318-b4f3-4101c02865f5" containerName="main" Nov 03 19:27:22.516142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.516113 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.615624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.615589 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-webhook-cert\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.615624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.615627 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8dgn\" (UniqueName: \"kubernetes.io/projected/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-kube-api-access-t8dgn\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.615858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.615656 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-metrics-certs\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.620915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.620883 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mzt96\"" Nov 03 19:27:22.620915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.620897 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 19:27:22.620915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.620884 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 19:27:22.621152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.621066 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 19:27:22.623717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.623676 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68f76587d6-bzh4v"] Nov 03 19:27:22.716447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.716405 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-webhook-cert\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.716447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.716449 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8dgn\" (UniqueName: \"kubernetes.io/projected/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-kube-api-access-t8dgn\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.716764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.716479 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-metrics-certs\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.719159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.719132 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-webhook-cert\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.719282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.719132 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-metrics-certs\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.752985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.752948 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" path="/var/lib/kubelet/pods/9aaae9b4-e7dc-4cdb-bb80-9a4639354338/volumes" Nov 03 19:27:22.779467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.779387 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8dgn\" (UniqueName: \"kubernetes.io/projected/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-kube-api-access-t8dgn\") pod \"maas-controller-68f76587d6-bzh4v\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:22.830334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:22.830287 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:23.215648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:23.215625 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68f76587d6-bzh4v"] Nov 03 19:27:23.217196 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:27:23.217168 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba59c1c_d9ed_428f_82ca_5d474390fdf0.slice/crio-95342f0fc3a810737f3eeb4e788d3a676070ff810e5fa7059818da6ac8d45850 WatchSource:0}: Error finding container 95342f0fc3a810737f3eeb4e788d3a676070ff810e5fa7059818da6ac8d45850: Status 404 returned error can't find the container with id 95342f0fc3a810737f3eeb4e788d3a676070ff810e5fa7059818da6ac8d45850 Nov 03 19:27:23.472166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:23.472069 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-bzh4v" event={"ID":"2ba59c1c-d9ed-428f-82ca-5d474390fdf0","Type":"ContainerStarted","Data":"95342f0fc3a810737f3eeb4e788d3a676070ff810e5fa7059818da6ac8d45850"} Nov 03 19:27:24.482285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:24.482243 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-bzh4v" event={"ID":"2ba59c1c-d9ed-428f-82ca-5d474390fdf0","Type":"ContainerStarted","Data":"3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812"} Nov 03 19:27:24.482879 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:24.482363 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:24.602746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:24.602664 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68f76587d6-bzh4v" podStartSLOduration=1.684188979 podStartE2EDuration="2.602645027s" totalImagesPullingTime="918.456048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:24.60193884 +0000 UTC m=+2974.084934759" watchObservedRunningTime="2026-11-03 19:27:24.602645027 +0000 UTC m=+2974.085640941" Nov 03 19:27:32.792823 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:27:32.792784 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:27:35.491501 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:35.491470 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:39.446219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.446174 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b"] Nov 03 19:27:39.446694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.446680 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aaae9b4-e7dc-4cdb-bb80-9a4639354338" containerName="manager" Nov 03 19:27:39.449917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.449903 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.578809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.578757 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.578809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.578812 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzwp\" (UniqueName: \"kubernetes.io/projected/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-kube-api-access-mhzwp\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.579096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.578850 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.579096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.578903 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.579096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.578954 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.579096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.578997 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.579096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.579024 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.579096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.579058 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.579096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.579083 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.656043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.656005 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b"] Nov 03 19:27:39.680228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680194 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680247 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680295 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680329 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680374 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680407 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680444 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680471 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzwp\" (UniqueName: \"kubernetes.io/projected/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-kube-api-access-mhzwp\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680642 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680845 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680896 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.680938 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.680895 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.681102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.681015 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.681297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.681271 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.683003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.682976 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.683115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.683040 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.743159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.743083 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.743447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.743420 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzwp\" (UniqueName: \"kubernetes.io/projected/4f2486e1-e5ca-46ac-aea5-9101cc6eb10f-kube-api-access-mhzwp\") pod \"maas-default-gateway-openshift-default-7854949c9-8pc6b\" (UID: \"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:39.762504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:39.762464 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:40.199560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.199527 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b"] Nov 03 19:27:40.230013 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:27:40.229984 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2486e1_e5ca_46ac_aea5_9101cc6eb10f.slice/crio-ad782ab6c67040044c226982cf65aafc868286e8d6b088b4b34ca6ddc05d1cdf WatchSource:0}: Error finding container ad782ab6c67040044c226982cf65aafc868286e8d6b088b4b34ca6ddc05d1cdf: Status 404 returned error can't find the container with id ad782ab6c67040044c226982cf65aafc868286e8d6b088b4b34ca6ddc05d1cdf Nov 03 19:27:40.232195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.232163 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:27:40.232300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.232233 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:27:40.232300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.232295 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:27:40.582504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.582401 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" event={"ID":"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f","Type":"ContainerStarted","Data":"7945dd0416cae83f72bbfc42c000e53f73a305079e9a4c28362ef50d8c79a790"} Nov 03 19:27:40.582504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.582460 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" event={"ID":"4f2486e1-e5ca-46ac-aea5-9101cc6eb10f","Type":"ContainerStarted","Data":"ad782ab6c67040044c226982cf65aafc868286e8d6b088b4b34ca6ddc05d1cdf"} Nov 03 19:27:40.707335 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.707278 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podStartSLOduration=1.70725963 podStartE2EDuration="1.70725963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:40.70548853 +0000 UTC m=+2990.188484445" watchObservedRunningTime="2026-11-03 19:27:40.70725963 +0000 UTC m=+2990.190255548" Nov 03 19:27:40.775978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:40.775939 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:41.772722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:41.772462 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 19:27:41.772722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:41.772531 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 19:27:42.775730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:42.774809 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 19:27:42.775730 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:42.774873 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 19:27:43.771154 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:43.771110 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 19:27:43.771503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:43.771462 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 19:27:44.382382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.382340 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x"] Nov 03 19:27:44.386263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.386244 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.435311 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.435273 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x"] Nov 03 19:27:44.483094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.483049 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:27:44.483094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.483050 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:27:44.483324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.483101 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:27:44.492622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.492597 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:27:44.529038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.529007 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.529202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.529043 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb8ml\" (UniqueName: \"kubernetes.io/projected/b15a590f-8e60-4871-b57a-09ed217f5c36-kube-api-access-fb8ml\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.529280 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.529213 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.630587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.630498 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.630835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.630606 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb8ml\" (UniqueName: \"kubernetes.io/projected/b15a590f-8e60-4871-b57a-09ed217f5c36-kube-api-access-fb8ml\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.630835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.630726 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.631336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.631315 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.632921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.632878 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.693966 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.693930 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb8ml\" (UniqueName: \"kubernetes.io/projected/b15a590f-8e60-4871-b57a-09ed217f5c36-kube-api-access-fb8ml\") pod \"maas-api-revoke-sub-85237e45-vcn2x\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.702207 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.702181 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:44.778876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.773859 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 19:27:44.778876 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:44.773919 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 19:27:45.102673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:45.102641 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x"] Nov 03 19:27:45.103939 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:27:45.103911 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15a590f_8e60_4871_b57a_09ed217f5c36.slice/crio-7ad0cab387af99767d2ad7b973b94adfc7e1b4039bb031792b8fee994e092b39 WatchSource:0}: Error finding container 7ad0cab387af99767d2ad7b973b94adfc7e1b4039bb031792b8fee994e092b39: Status 404 returned error can't find the container with id 7ad0cab387af99767d2ad7b973b94adfc7e1b4039bb031792b8fee994e092b39 Nov 03 19:27:45.615232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:45.615192 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" event={"ID":"b15a590f-8e60-4871-b57a-09ed217f5c36","Type":"ContainerStarted","Data":"abe3ac23eeadb7ada771fc5a37181aa80016600078217c100d40cf0561f6b4cc"} Nov 03 19:27:45.615685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:45.615260 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" event={"ID":"b15a590f-8e60-4871-b57a-09ed217f5c36","Type":"ContainerStarted","Data":"7ad0cab387af99767d2ad7b973b94adfc7e1b4039bb031792b8fee994e092b39"} Nov 03 19:27:45.742427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:45.742373 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" podStartSLOduration=1.74235954 podStartE2EDuration="1.74235954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:45.740075253 +0000 UTC m=+2995.223071168" watchObservedRunningTime="2026-11-03 19:27:45.74235954 +0000 UTC m=+2995.225355495" Nov 03 19:27:45.763098 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:45.763060 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 19:27:45.763369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:45.763339 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 19:27:46.622598 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:46.622556 4948 generic.go:356] "Generic (PLEG): container finished" podID="b15a590f-8e60-4871-b57a-09ed217f5c36" containerID="abe3ac23eeadb7ada771fc5a37181aa80016600078217c100d40cf0561f6b4cc" exitCode=0 Nov 03 19:27:46.623019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:46.622637 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" event={"ID":"b15a590f-8e60-4871-b57a-09ed217f5c36","Type":"ContainerDied","Data":"abe3ac23eeadb7ada771fc5a37181aa80016600078217c100d40cf0561f6b4cc"} Nov 03 19:27:46.763819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:46.763778 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 19:27:46.764004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:46.763835 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 19:27:47.762902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.762867 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 19:27:47.763347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.763313 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 19:27:47.779471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.779443 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:47.864051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.864018 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15a590f-8e60-4871-b57a-09ed217f5c36-kube-api-access-fb8ml\" (UniqueName: \"kubernetes.io/projected/b15a590f-8e60-4871-b57a-09ed217f5c36-kube-api-access-fb8ml\") pod \"b15a590f-8e60-4871-b57a-09ed217f5c36\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " Nov 03 19:27:47.864286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.864130 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-xks-ca\") pod \"b15a590f-8e60-4871-b57a-09ed217f5c36\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " Nov 03 19:27:47.864286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.864192 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-openshift-ca\") pod \"b15a590f-8e60-4871-b57a-09ed217f5c36\" (UID: \"b15a590f-8e60-4871-b57a-09ed217f5c36\") " Nov 03 19:27:47.864606 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.864568 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-openshift-ca" pod "b15a590f-8e60-4871-b57a-09ed217f5c36" (UID: "b15a590f-8e60-4871-b57a-09ed217f5c36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:47.865435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.865409 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:47.866796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.866769 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15a590f-8e60-4871-b57a-09ed217f5c36-kube-api-access-fb8ml" pod "b15a590f-8e60-4871-b57a-09ed217f5c36" (UID: "b15a590f-8e60-4871-b57a-09ed217f5c36"). InnerVolumeSpecName "kube-api-access-fb8ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:47.866891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.866769 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-xks-ca" pod "b15a590f-8e60-4871-b57a-09ed217f5c36" (UID: "b15a590f-8e60-4871-b57a-09ed217f5c36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:47.966166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.966127 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb8ml\" (UniqueName: \"kubernetes.io/projected/b15a590f-8e60-4871-b57a-09ed217f5c36-kube-api-access-fb8ml\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:47.966166 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:47.966166 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15a590f-8e60-4871-b57a-09ed217f5c36-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:48.635106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:48.635076 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" Nov 03 19:27:48.635299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:48.635113 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x" event={"ID":"b15a590f-8e60-4871-b57a-09ed217f5c36","Type":"ContainerDied","Data":"7ad0cab387af99767d2ad7b973b94adfc7e1b4039bb031792b8fee994e092b39"} Nov 03 19:27:48.635299 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:48.635161 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad0cab387af99767d2ad7b973b94adfc7e1b4039bb031792b8fee994e092b39" Nov 03 19:27:48.767567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:48.767533 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:48.768090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:48.767942 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:48.768778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:48.768756 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" Nov 03 19:27:54.418502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.418468 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d"] Nov 03 19:27:54.419115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.419094 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b15a590f-8e60-4871-b57a-09ed217f5c36" containerName="revoke-keys" Nov 03 19:27:54.419164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.419119 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15a590f-8e60-4871-b57a-09ed217f5c36" containerName="revoke-keys" Nov 03 19:27:54.419212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.419202 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15a590f-8e60-4871-b57a-09ed217f5c36" containerName="revoke-keys" Nov 03 19:27:54.422403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.422383 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.521837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.521798 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d"] Nov 03 19:27:54.528942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.528915 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.528951 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.528973 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd296b3-a365-49a1-8fe2-c94e943a7618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.529051 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.529142 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.529195 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.529267 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.529296 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.529456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.529369 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxkd8\" (UniqueName: \"kubernetes.io/projected/cdd296b3-a365-49a1-8fe2-c94e943a7618-kube-api-access-kxkd8\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.630664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630621 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.630664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630673 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.630921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630727 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.630921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630757 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.630921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630805 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxkd8\" (UniqueName: \"kubernetes.io/projected/cdd296b3-a365-49a1-8fe2-c94e943a7618-kube-api-access-kxkd8\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.630921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630875 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.630921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630901 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.631206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630928 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd296b3-a365-49a1-8fe2-c94e943a7618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.631206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.630955 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.631206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.631108 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.631206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.631148 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.631427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.631358 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.631494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.631473 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.631779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.631744 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd296b3-a365-49a1-8fe2-c94e943a7618-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.633284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.633258 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.633553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.633529 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.693348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.693313 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd296b3-a365-49a1-8fe2-c94e943a7618-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.693509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.693395 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxkd8\" (UniqueName: \"kubernetes.io/projected/cdd296b3-a365-49a1-8fe2-c94e943a7618-kube-api-access-kxkd8\") pod \"maas-default-gateway-openshift-default-7854949c9-tg88d\" (UID: \"cdd296b3-a365-49a1-8fe2-c94e943a7618\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:54.735001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:54.734968 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:55.169483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.169446 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d"] Nov 03 19:27:55.170898 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:27:55.170872 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd296b3_a365_49a1_8fe2_c94e943a7618.slice/crio-b4e632c2bff1f2fec1971e393cc210846af853b203b8456ae7019764221b835c WatchSource:0}: Error finding container b4e632c2bff1f2fec1971e393cc210846af853b203b8456ae7019764221b835c: Status 404 returned error can't find the container with id b4e632c2bff1f2fec1971e393cc210846af853b203b8456ae7019764221b835c Nov 03 19:27:55.173192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.173160 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:27:55.173293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.173225 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:27:55.173293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.173254 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:27:55.686225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.686186 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" event={"ID":"cdd296b3-a365-49a1-8fe2-c94e943a7618","Type":"ContainerStarted","Data":"958185c12eef4540f546725e270e43e57eac026e1f5e8a0030671522541247aa"} Nov 03 19:27:55.686642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.686257 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" event={"ID":"cdd296b3-a365-49a1-8fe2-c94e943a7618","Type":"ContainerStarted","Data":"b4e632c2bff1f2fec1971e393cc210846af853b203b8456ae7019764221b835c"} Nov 03 19:27:55.710358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.710332 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:27:55.717286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.717265 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:27:55.735596 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.735567 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:27:55.817691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.817636 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" podStartSLOduration=1.8176230850000001 podStartE2EDuration="1.817623085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:55.815267422 +0000 UTC m=+3005.298263336" watchObservedRunningTime="2026-11-03 19:27:55.817623085 +0000 UTC m=+3005.300618993" Nov 03 19:27:55.977694 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.977609 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:27:55.985314 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:55.985290 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:27:56.737227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:56.737194 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 19:27:56.737642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:56.737264 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" podUID="cdd296b3-a365-49a1-8fe2-c94e943a7618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 19:27:57.117483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.117393 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68f76587d6-bzh4v"] Nov 03 19:27:57.117681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.117625 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68f76587d6-bzh4v" podUID="2ba59c1c-d9ed-428f-82ca-5d474390fdf0" containerName="manager" containerID="cri-o://3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812" gracePeriod=30 Nov 03 19:27:57.508885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.508861 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:57.686801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.685925 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-webhook-cert\") pod \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " Nov 03 19:27:57.686801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.685974 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-kube-api-access-t8dgn\" (UniqueName: \"kubernetes.io/projected/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-kube-api-access-t8dgn\") pod \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " Nov 03 19:27:57.686801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.686125 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-metrics-certs\") pod \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\" (UID: \"2ba59c1c-d9ed-428f-82ca-5d474390fdf0\") " Nov 03 19:27:57.692733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.689659 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-kube-api-access-t8dgn" pod "2ba59c1c-d9ed-428f-82ca-5d474390fdf0" (UID: "2ba59c1c-d9ed-428f-82ca-5d474390fdf0"). InnerVolumeSpecName "kube-api-access-t8dgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:57.692733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.692719 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-metrics-certs" pod "2ba59c1c-d9ed-428f-82ca-5d474390fdf0" (UID: "2ba59c1c-d9ed-428f-82ca-5d474390fdf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:57.692935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.692201 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-webhook-cert" pod "2ba59c1c-d9ed-428f-82ca-5d474390fdf0" (UID: "2ba59c1c-d9ed-428f-82ca-5d474390fdf0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:57.708450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.708417 4948 generic.go:356] "Generic (PLEG): container finished" podID="2ba59c1c-d9ed-428f-82ca-5d474390fdf0" containerID="3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812" exitCode=0 Nov 03 19:27:57.709182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.709146 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-bzh4v" event={"ID":"2ba59c1c-d9ed-428f-82ca-5d474390fdf0","Type":"ContainerDied","Data":"3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812"} Nov 03 19:27:57.709309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.709205 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-bzh4v" event={"ID":"2ba59c1c-d9ed-428f-82ca-5d474390fdf0","Type":"ContainerDied","Data":"95342f0fc3a810737f3eeb4e788d3a676070ff810e5fa7059818da6ac8d45850"} Nov 03 19:27:57.709309 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.709232 4948 scope.go:122] "RemoveContainer" containerID="3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812" Nov 03 19:27:57.709455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.709437 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-bzh4v" Nov 03 19:27:57.724992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.724974 4948 scope.go:122] "RemoveContainer" containerID="3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812" Nov 03 19:27:57.725304 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:27:57.725277 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812\": container with ID starting with 3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812 not found: ID does not exist" containerID="3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812" Nov 03 19:27:57.725356 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.725319 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812"} err="failed to get container status \"3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812\": rpc error: code = NotFound desc = could not find container \"3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812\": container with ID starting with 3e33d86bc1085ad9fc94d3a554fb9547188f6354eb17b0d9841fc40731a1f812 not found: ID does not exist" Nov 03 19:27:57.738128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.735871 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 19:27:57.738128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.735928 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" podUID="cdd296b3-a365-49a1-8fe2-c94e943a7618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 19:27:57.787791 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.787760 4948 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-webhook-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:57.787791 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.787791 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8dgn\" (UniqueName: \"kubernetes.io/projected/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-kube-api-access-t8dgn\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:57.788056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.787809 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba59c1c-d9ed-428f-82ca-5d474390fdf0-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:57.892864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.892830 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68f76587d6-bzh4v"] Nov 03 19:27:57.896076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:57.896055 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68f76587d6-bzh4v"] Nov 03 19:27:58.735782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:58.735747 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 19:27:58.735969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:58.735799 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" podUID="cdd296b3-a365-49a1-8fe2-c94e943a7618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 19:27:58.775995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:58.775957 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba59c1c-d9ed-428f-82ca-5d474390fdf0" path="/var/lib/kubelet/pods/2ba59c1c-d9ed-428f-82ca-5d474390fdf0/volumes" Nov 03 19:27:59.308235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.308198 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68f76587d6-t8kgt"] Nov 03 19:27:59.308637 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.308624 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2ba59c1c-d9ed-428f-82ca-5d474390fdf0" containerName="manager" Nov 03 19:27:59.308691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.308638 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba59c1c-d9ed-428f-82ca-5d474390fdf0" containerName="manager" Nov 03 19:27:59.308746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.308723 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba59c1c-d9ed-428f-82ca-5d474390fdf0" containerName="manager" Nov 03 19:27:59.313234 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.313212 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.374951 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.374920 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 19:27:59.400941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.400910 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tmg4\" (UniqueName: \"kubernetes.io/projected/5feb0c16-6030-4cda-8e9f-243c7a77d023-kube-api-access-2tmg4\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.401176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.401161 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feb0c16-6030-4cda-8e9f-243c7a77d023-webhook-cert\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.401316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.401304 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feb0c16-6030-4cda-8e9f-243c7a77d023-metrics-certs\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.425371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.425275 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 19:27:59.425371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.425333 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mzt96\"" Nov 03 19:27:59.425616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.425521 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 19:27:59.428263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.428237 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68f76587d6-t8kgt"] Nov 03 19:27:59.501983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.501949 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feb0c16-6030-4cda-8e9f-243c7a77d023-metrics-certs\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.502193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.502009 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tmg4\" (UniqueName: \"kubernetes.io/projected/5feb0c16-6030-4cda-8e9f-243c7a77d023-kube-api-access-2tmg4\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.502193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.502067 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feb0c16-6030-4cda-8e9f-243c7a77d023-webhook-cert\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.504621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.504594 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feb0c16-6030-4cda-8e9f-243c7a77d023-metrics-certs\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.504779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.504672 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feb0c16-6030-4cda-8e9f-243c7a77d023-webhook-cert\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.577771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.577675 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tmg4\" (UniqueName: \"kubernetes.io/projected/5feb0c16-6030-4cda-8e9f-243c7a77d023-kube-api-access-2tmg4\") pod \"maas-controller-68f76587d6-t8kgt\" (UID: \"5feb0c16-6030-4cda-8e9f-243c7a77d023\") " pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.627247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.627210 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:27:59.736133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.736099 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 19:27:59.736403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:27:59.736162 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" podUID="cdd296b3-a365-49a1-8fe2-c94e943a7618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 19:28:00.029027 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:00.028998 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68f76587d6-t8kgt"] Nov 03 19:28:00.029471 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:28:00.029445 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5feb0c16_6030_4cda_8e9f_243c7a77d023.slice/crio-3efa7be1998625f679c1c76d17f4c79becf8e62e4049317c517c6f02fc31a77d WatchSource:0}: Error finding container 3efa7be1998625f679c1c76d17f4c79becf8e62e4049317c517c6f02fc31a77d: Status 404 returned error can't find the container with id 3efa7be1998625f679c1c76d17f4c79becf8e62e4049317c517c6f02fc31a77d Nov 03 19:28:00.733191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:00.733109 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-t8kgt" event={"ID":"5feb0c16-6030-4cda-8e9f-243c7a77d023","Type":"ContainerStarted","Data":"3efa7be1998625f679c1c76d17f4c79becf8e62e4049317c517c6f02fc31a77d"} Nov 03 19:28:00.735388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:00.735365 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 19:28:00.735460 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:00.735411 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" podUID="cdd296b3-a365-49a1-8fe2-c94e943a7618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 19:28:01.735640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:01.735603 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-tg88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 19:28:01.736128 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:01.735654 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" podUID="cdd296b3-a365-49a1-8fe2-c94e943a7618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 19:28:01.742617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:01.742587 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68f76587d6-t8kgt" event={"ID":"5feb0c16-6030-4cda-8e9f-243c7a77d023","Type":"ContainerStarted","Data":"832b658332b51f295a8b87145048a8d46b2a1c9cd9c9cdb4ddc2e3c170411444"} Nov 03 19:28:01.742764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:01.742673 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:28:01.913717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:01.913654 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68f76587d6-t8kgt" podStartSLOduration=1.910051722 podStartE2EDuration="2.913639622s" totalImagesPullingTime="1.0035879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:28:01.911602876 +0000 UTC m=+3011.394598792" watchObservedRunningTime="2026-11-03 19:28:01.913639622 +0000 UTC m=+3011.396635540" Nov 03 19:28:02.796674 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:28:02.796623 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:28:03.433909 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:03.433874 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:28:03.434267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:03.434230 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:28:03.435108 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:03.435088 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-tg88d" Nov 03 19:28:12.752798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:12.752762 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68f76587d6-t8kgt" Nov 03 19:28:32.792075 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:28:32.792031 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:28:53.221437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.221404 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz"] Nov 03 19:28:53.224858 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.224842 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.280648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.280601 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz"] Nov 03 19:28:53.298831 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.298795 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.299017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.298850 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.299017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.298923 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdktj\" (UniqueName: \"kubernetes.io/projected/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-kube-api-access-wdktj\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.328305 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.328273 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:28:53.328450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.328311 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:28:53.328450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.328311 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:28:53.329228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.329205 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:28:53.400205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.400172 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.400367 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.400209 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdktj\" (UniqueName: \"kubernetes.io/projected/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-kube-api-access-wdktj\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.400367 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.400342 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.401014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.400988 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.402642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.402618 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.462868 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.462840 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdktj\" (UniqueName: \"kubernetes.io/projected/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-kube-api-access-wdktj\") pod \"maas-api-revoke-sub-c1e271a9-h6zjz\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.537813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.537732 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:53.914714 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:53.914677 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz"] Nov 03 19:28:53.916158 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:28:53.916124 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d9b1cc_da85_47aa_baf2_47b9e9d7f78f.slice/crio-b2feddaa74439f8ded24219048d87d1b6188b11e54a1b6943010e49394846921 WatchSource:0}: Error finding container b2feddaa74439f8ded24219048d87d1b6188b11e54a1b6943010e49394846921: Status 404 returned error can't find the container with id b2feddaa74439f8ded24219048d87d1b6188b11e54a1b6943010e49394846921 Nov 03 19:28:54.080945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:54.080910 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" event={"ID":"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f","Type":"ContainerStarted","Data":"b2feddaa74439f8ded24219048d87d1b6188b11e54a1b6943010e49394846921"} Nov 03 19:28:55.087802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:55.087760 4948 generic.go:356] "Generic (PLEG): container finished" podID="77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" containerID="ef0fb517ace09ea0172c7a155726ddedf958a752b92afdfd1c842ebb67aaed21" exitCode=0 Nov 03 19:28:55.088217 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:55.087846 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" event={"ID":"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f","Type":"ContainerDied","Data":"ef0fb517ace09ea0172c7a155726ddedf958a752b92afdfd1c842ebb67aaed21"} Nov 03 19:28:56.230784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.230756 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:56.326516 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.326483 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-kube-api-access-wdktj\" (UniqueName: \"kubernetes.io/projected/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-kube-api-access-wdktj\") pod \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " Nov 03 19:28:56.326759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.326567 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-openshift-ca\") pod \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " Nov 03 19:28:56.326759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.326653 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-xks-ca\") pod \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\" (UID: \"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f\") " Nov 03 19:28:56.326977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.326952 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-openshift-ca" pod "77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" (UID: "77d9b1cc-da85-47aa-baf2-47b9e9d7f78f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:28:56.328753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.328727 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-xks-ca" pod "77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" (UID: "77d9b1cc-da85-47aa-baf2-47b9e9d7f78f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:28:56.328839 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.328819 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-kube-api-access-wdktj" pod "77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" (UID: "77d9b1cc-da85-47aa-baf2-47b9e9d7f78f"). InnerVolumeSpecName "kube-api-access-wdktj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:28:56.428018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.427938 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:56.428018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.427966 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:56.428018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:56.427976 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdktj\" (UniqueName: \"kubernetes.io/projected/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f-kube-api-access-wdktj\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:28:57.101814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:57.101728 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" Nov 03 19:28:57.101814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:57.101750 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz" event={"ID":"77d9b1cc-da85-47aa-baf2-47b9e9d7f78f","Type":"ContainerDied","Data":"b2feddaa74439f8ded24219048d87d1b6188b11e54a1b6943010e49394846921"} Nov 03 19:28:57.101814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:28:57.101798 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2feddaa74439f8ded24219048d87d1b6188b11e54a1b6943010e49394846921" Nov 03 19:29:02.792657 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:29:02.792616 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:29:09.434823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.434785 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c"] Nov 03 19:29:09.435238 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.435223 4948 cpu_manager.go:414] "containerMap: removing container" podUID="77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" containerName="revoke-keys" Nov 03 19:29:09.435293 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.435239 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" containerName="revoke-keys" Nov 03 19:29:09.435338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.435329 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" containerName="revoke-keys" Nov 03 19:29:09.438797 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.438780 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.546391 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.546362 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c"] Nov 03 19:29:09.558695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.558664 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.558857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.558758 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.558920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.558890 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.558976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.558955 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5a14d7-8474-45ff-ae36-87c0aabad985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.559016 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.558994 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.559085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.559066 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.559131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.559120 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dh2r\" (UniqueName: \"kubernetes.io/projected/ab5a14d7-8474-45ff-ae36-87c0aabad985-kube-api-access-6dh2r\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.559192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.559177 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.559240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.559221 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660000 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dh2r\" (UniqueName: \"kubernetes.io/projected/ab5a14d7-8474-45ff-ae36-87c0aabad985-kube-api-access-6dh2r\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660064 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660116 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660161 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660204 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660233 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660271 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5a14d7-8474-45ff-ae36-87c0aabad985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660311 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660564 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660359 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660567 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660774 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660650 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660904 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660779 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.660997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.660970 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.661063 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.661011 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5a14d7-8474-45ff-ae36-87c0aabad985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.663010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.662977 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.663269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.663248 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.722536 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.722463 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5a14d7-8474-45ff-ae36-87c0aabad985-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.722805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.722782 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dh2r\" (UniqueName: \"kubernetes.io/projected/ab5a14d7-8474-45ff-ae36-87c0aabad985-kube-api-access-6dh2r\") pod \"maas-default-gateway-openshift-default-7854949c9-fmq4c\" (UID: \"ab5a14d7-8474-45ff-ae36-87c0aabad985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:09.751906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:09.751872 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:10.191223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:10.189924 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c"] Nov 03 19:29:10.194422 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:29:10.194394 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5a14d7_8474_45ff_ae36_87c0aabad985.slice/crio-897d718fc2f70157fcb616332a1538216bace28b8c90abade6987762ffc49efe WatchSource:0}: Error finding container 897d718fc2f70157fcb616332a1538216bace28b8c90abade6987762ffc49efe: Status 404 returned error can't find the container with id 897d718fc2f70157fcb616332a1538216bace28b8c90abade6987762ffc49efe Nov 03 19:29:10.196492 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:10.196462 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:29:10.196560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:10.196525 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:29:10.196560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:10.196552 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:29:11.193636 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:11.193599 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" event={"ID":"ab5a14d7-8474-45ff-ae36-87c0aabad985","Type":"ContainerStarted","Data":"2c0a0db3508a616368ee436a82b28b8f2473c8e6842aac2e9cc851acf9e9b67f"} Nov 03 19:29:11.194078 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:11.193657 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" event={"ID":"ab5a14d7-8474-45ff-ae36-87c0aabad985","Type":"ContainerStarted","Data":"897d718fc2f70157fcb616332a1538216bace28b8c90abade6987762ffc49efe"} Nov 03 19:29:11.423321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:11.423263 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podStartSLOduration=2.423248114 podStartE2EDuration="2.423248114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:29:11.371184264 +0000 UTC m=+3080.854180181" watchObservedRunningTime="2026-11-03 19:29:11.423248114 +0000 UTC m=+3080.906244029" Nov 03 19:29:11.769724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:11.758730 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:11.769724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:11.763413 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 19:29:11.769724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:11.763572 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 19:29:12.774466 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:12.774429 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 19:29:12.775157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:12.775122 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 19:29:13.767723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:13.760256 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 19:29:13.767723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:13.760311 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 19:29:14.776215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:14.776162 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 19:29:14.776805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:14.776232 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 19:29:15.771205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:15.771164 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 19:29:15.771507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:15.771476 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 19:29:16.752342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:16.752302 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 19:29:16.752762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:16.752372 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 19:29:17.752620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:17.752587 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 19:29:17.753094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:17.752647 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 19:29:19.792957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:19.792904 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:29:19.793508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:19.792975 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:29:20.615818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:20.615783 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:20.616142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:20.616118 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:20.616935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:20.616916 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" Nov 03 19:29:24.827914 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.827880 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t"] Nov 03 19:29:24.831982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.831957 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.885502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.885471 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:29:24.887169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.887136 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.887341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.887299 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.887420 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.887342 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npfvs\" (UniqueName: \"kubernetes.io/projected/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-kube-api-access-npfvs\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.887789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.887767 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t"] Nov 03 19:29:24.934828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.934789 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:29:24.934828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.934811 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:29:24.934828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.934801 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:29:24.988273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.988238 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.988446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.988278 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npfvs\" (UniqueName: \"kubernetes.io/projected/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-kube-api-access-npfvs\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.988446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.988405 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.988888 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.988866 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:24.990590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:24.990573 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:25.050943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:25.050917 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npfvs\" (UniqueName: \"kubernetes.io/projected/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-kube-api-access-npfvs\") pod \"maas-api-revoke-sub-2e1f5b5f-wmv7t\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:25.145551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:25.145484 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:25.523983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:25.523957 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t"] Nov 03 19:29:25.525787 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:29:25.525688 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce49bb21_6e64_4b25_98f2_9bd397d6eb91.slice/crio-815d9024eee6eaef62e60a515d5c982ec536ca5faedcce6a284bc19606c43db0 WatchSource:0}: Error finding container 815d9024eee6eaef62e60a515d5c982ec536ca5faedcce6a284bc19606c43db0: Status 404 returned error can't find the container with id 815d9024eee6eaef62e60a515d5c982ec536ca5faedcce6a284bc19606c43db0 Nov 03 19:29:26.356146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:26.356110 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" event={"ID":"ce49bb21-6e64-4b25-98f2-9bd397d6eb91","Type":"ContainerStarted","Data":"22ff37a4bbece740425e184098c429515ff5ed91b939081225efc18ac6e925aa"} Nov 03 19:29:26.356523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:26.356185 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" event={"ID":"ce49bb21-6e64-4b25-98f2-9bd397d6eb91","Type":"ContainerStarted","Data":"815d9024eee6eaef62e60a515d5c982ec536ca5faedcce6a284bc19606c43db0"} Nov 03 19:29:26.477815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:26.477758 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" podStartSLOduration=2.47774401 podStartE2EDuration="2.47774401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:29:26.475011227 +0000 UTC m=+3095.958007154" watchObservedRunningTime="2026-11-03 19:29:26.47774401 +0000 UTC m=+3095.960739921" Nov 03 19:29:27.364165 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:27.364127 4948 generic.go:356] "Generic (PLEG): container finished" podID="ce49bb21-6e64-4b25-98f2-9bd397d6eb91" containerID="22ff37a4bbece740425e184098c429515ff5ed91b939081225efc18ac6e925aa" exitCode=0 Nov 03 19:29:27.364591 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:27.364212 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" event={"ID":"ce49bb21-6e64-4b25-98f2-9bd397d6eb91","Type":"ContainerDied","Data":"22ff37a4bbece740425e184098c429515ff5ed91b939081225efc18ac6e925aa"} Nov 03 19:29:28.502253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.502227 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:28.512808 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.512782 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-openshift-ca\") pod \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " Nov 03 19:29:28.512960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.512853 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-kube-api-access-npfvs\" (UniqueName: \"kubernetes.io/projected/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-kube-api-access-npfvs\") pod \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " Nov 03 19:29:28.512960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.512889 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-xks-ca\") pod \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\" (UID: \"ce49bb21-6e64-4b25-98f2-9bd397d6eb91\") " Nov 03 19:29:28.513127 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.513081 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-openshift-ca" pod "ce49bb21-6e64-4b25-98f2-9bd397d6eb91" (UID: "ce49bb21-6e64-4b25-98f2-9bd397d6eb91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:29:28.513243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.513223 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:28.514830 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.514809 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-xks-ca" pod "ce49bb21-6e64-4b25-98f2-9bd397d6eb91" (UID: "ce49bb21-6e64-4b25-98f2-9bd397d6eb91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:29:28.515023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.515007 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-kube-api-access-npfvs" pod "ce49bb21-6e64-4b25-98f2-9bd397d6eb91" (UID: "ce49bb21-6e64-4b25-98f2-9bd397d6eb91"). InnerVolumeSpecName "kube-api-access-npfvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:29:28.614316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.614288 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npfvs\" (UniqueName: \"kubernetes.io/projected/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-kube-api-access-npfvs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:28.614316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:28.614314 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce49bb21-6e64-4b25-98f2-9bd397d6eb91-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:29.377567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:29.377535 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" Nov 03 19:29:29.377809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:29.377567 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t" event={"ID":"ce49bb21-6e64-4b25-98f2-9bd397d6eb91","Type":"ContainerDied","Data":"815d9024eee6eaef62e60a515d5c982ec536ca5faedcce6a284bc19606c43db0"} Nov 03 19:29:29.377809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:29.377615 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815d9024eee6eaef62e60a515d5c982ec536ca5faedcce6a284bc19606c43db0" Nov 03 19:29:32.791605 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:29:32.791529 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:29:37.192789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.192740 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h"] Nov 03 19:29:37.193289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.193201 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ce49bb21-6e64-4b25-98f2-9bd397d6eb91" containerName="revoke-keys" Nov 03 19:29:37.193289 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.193214 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce49bb21-6e64-4b25-98f2-9bd397d6eb91" containerName="revoke-keys" Nov 03 19:29:37.193434 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.193323 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce49bb21-6e64-4b25-98f2-9bd397d6eb91" containerName="revoke-keys" Nov 03 19:29:37.197244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.197213 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.250988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.250952 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h"] Nov 03 19:29:37.297658 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.297621 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:29:37.297658 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.297630 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:29:37.297890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.297736 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:29:37.299159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.299142 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.299227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.299167 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7668s\" (UniqueName: \"kubernetes.io/projected/3b8d90cf-8396-4840-a75b-34e54478311a-kube-api-access-7668s\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.299330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.299312 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.301421 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.301402 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:29:37.400524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.400491 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.400690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.400585 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.400690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.400603 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7668s\" (UniqueName: \"kubernetes.io/projected/3b8d90cf-8396-4840-a75b-34e54478311a-kube-api-access-7668s\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.401215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.401191 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.402975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.402953 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.465152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.465082 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7668s\" (UniqueName: \"kubernetes.io/projected/3b8d90cf-8396-4840-a75b-34e54478311a-kube-api-access-7668s\") pod \"maas-api-revoke-sub-ec843018-gg54h\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.517854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.517828 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:37.896439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:37.896346 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h"] Nov 03 19:29:37.913257 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:29:37.913212 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8d90cf_8396_4840_a75b_34e54478311a.slice/crio-0c73f2d0b9fa09de304cc33441f22462c9a7e184069295ba1e2dd2318268b808 WatchSource:0}: Error finding container 0c73f2d0b9fa09de304cc33441f22462c9a7e184069295ba1e2dd2318268b808: Status 404 returned error can't find the container with id 0c73f2d0b9fa09de304cc33441f22462c9a7e184069295ba1e2dd2318268b808 Nov 03 19:29:38.433528 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:38.433495 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" event={"ID":"3b8d90cf-8396-4840-a75b-34e54478311a","Type":"ContainerStarted","Data":"3a9545f8e193baecbbd6c685d9afc0f13f1126109d86566289ad0ed5b1dcf9bb"} Nov 03 19:29:38.433921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:38.433554 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" event={"ID":"3b8d90cf-8396-4840-a75b-34e54478311a","Type":"ContainerStarted","Data":"0c73f2d0b9fa09de304cc33441f22462c9a7e184069295ba1e2dd2318268b808"} Nov 03 19:29:38.552567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:38.552516 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" podStartSLOduration=1.552498097 podStartE2EDuration="1.552498097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:29:38.551194724 +0000 UTC m=+3108.034190662" watchObservedRunningTime="2026-11-03 19:29:38.552498097 +0000 UTC m=+3108.035494012" Nov 03 19:29:39.441689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:39.441651 4948 generic.go:356] "Generic (PLEG): container finished" podID="3b8d90cf-8396-4840-a75b-34e54478311a" containerID="3a9545f8e193baecbbd6c685d9afc0f13f1126109d86566289ad0ed5b1dcf9bb" exitCode=0 Nov 03 19:29:39.442134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:39.441740 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" event={"ID":"3b8d90cf-8396-4840-a75b-34e54478311a","Type":"ContainerDied","Data":"3a9545f8e193baecbbd6c685d9afc0f13f1126109d86566289ad0ed5b1dcf9bb"} Nov 03 19:29:40.593139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.593109 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:40.634638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.634607 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-openshift-ca\") pod \"3b8d90cf-8396-4840-a75b-34e54478311a\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " Nov 03 19:29:40.634815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.634693 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-xks-ca\") pod \"3b8d90cf-8396-4840-a75b-34e54478311a\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " Nov 03 19:29:40.634815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.634783 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8d90cf-8396-4840-a75b-34e54478311a-kube-api-access-7668s\" (UniqueName: \"kubernetes.io/projected/3b8d90cf-8396-4840-a75b-34e54478311a-kube-api-access-7668s\") pod \"3b8d90cf-8396-4840-a75b-34e54478311a\" (UID: \"3b8d90cf-8396-4840-a75b-34e54478311a\") " Nov 03 19:29:40.635057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.635031 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-openshift-ca" pod "3b8d90cf-8396-4840-a75b-34e54478311a" (UID: "3b8d90cf-8396-4840-a75b-34e54478311a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:29:40.636739 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.636714 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-xks-ca" pod "3b8d90cf-8396-4840-a75b-34e54478311a" (UID: "3b8d90cf-8396-4840-a75b-34e54478311a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:29:40.636826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.636803 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8d90cf-8396-4840-a75b-34e54478311a-kube-api-access-7668s" pod "3b8d90cf-8396-4840-a75b-34e54478311a" (UID: "3b8d90cf-8396-4840-a75b-34e54478311a"). InnerVolumeSpecName "kube-api-access-7668s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:29:40.736057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.735992 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:40.736057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.736016 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8d90cf-8396-4840-a75b-34e54478311a-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:40.736057 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:40.736027 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7668s\" (UniqueName: \"kubernetes.io/projected/3b8d90cf-8396-4840-a75b-34e54478311a-kube-api-access-7668s\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:29:41.456365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:41.456331 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" Nov 03 19:29:41.456551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:41.456365 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h" event={"ID":"3b8d90cf-8396-4840-a75b-34e54478311a","Type":"ContainerDied","Data":"0c73f2d0b9fa09de304cc33441f22462c9a7e184069295ba1e2dd2318268b808"} Nov 03 19:29:41.456551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:29:41.456409 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c73f2d0b9fa09de304cc33441f22462c9a7e184069295ba1e2dd2318268b808" Nov 03 19:30:00.163540 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.163444 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf"] Nov 03 19:30:00.164263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.163940 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3b8d90cf-8396-4840-a75b-34e54478311a" containerName="revoke-keys" Nov 03 19:30:00.164263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.163953 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8d90cf-8396-4840-a75b-34e54478311a" containerName="revoke-keys" Nov 03 19:30:00.164263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.164030 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8d90cf-8396-4840-a75b-34e54478311a" containerName="revoke-keys" Nov 03 19:30:00.167406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.167382 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.207722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.207679 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28424537-7cb3-4523-b228-d4d28c57986c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895570-zl6hf\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.207934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.207887 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgzm\" (UniqueName: \"kubernetes.io/projected/28424537-7cb3-4523-b228-d4d28c57986c-kube-api-access-zlgzm\") pod \"maas-api-key-cleanup-29895570-zl6hf\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.222430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.222396 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf"] Nov 03 19:30:00.252447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.252406 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:30:00.252605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.252459 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:30:00.269361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.269326 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:30:00.309029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.308993 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgzm\" (UniqueName: \"kubernetes.io/projected/28424537-7cb3-4523-b228-d4d28c57986c-kube-api-access-zlgzm\") pod \"maas-api-key-cleanup-29895570-zl6hf\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.309179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.309047 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28424537-7cb3-4523-b228-d4d28c57986c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895570-zl6hf\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.309761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.309733 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28424537-7cb3-4523-b228-d4d28c57986c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895570-zl6hf\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.371130 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.371095 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgzm\" (UniqueName: \"kubernetes.io/projected/28424537-7cb3-4523-b228-d4d28c57986c-kube-api-access-zlgzm\") pod \"maas-api-key-cleanup-29895570-zl6hf\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.481935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.481890 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:00.858600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:00.858575 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf"] Nov 03 19:30:00.859565 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:30:00.859539 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28424537_7cb3_4523_b228_d4d28c57986c.slice/crio-88642ab73ab5a6b4645868da214715a8579fc3849d8bb4f3009c8dfe532dd514 WatchSource:0}: Error finding container 88642ab73ab5a6b4645868da214715a8579fc3849d8bb4f3009c8dfe532dd514: Status 404 returned error can't find the container with id 88642ab73ab5a6b4645868da214715a8579fc3849d8bb4f3009c8dfe532dd514 Nov 03 19:30:01.620442 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:01.620407 4948 generic.go:356] "Generic (PLEG): container finished" podID="28424537-7cb3-4523-b228-d4d28c57986c" containerID="5f2202a048790fe05bdd4d6d98cfd65dca26ebc9f7964b90e09759c47d046d42" exitCode=60 Nov 03 19:30:01.620939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:01.620499 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" event={"ID":"28424537-7cb3-4523-b228-d4d28c57986c","Type":"ContainerDied","Data":"5f2202a048790fe05bdd4d6d98cfd65dca26ebc9f7964b90e09759c47d046d42"} Nov 03 19:30:01.620939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:01.620552 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" event={"ID":"28424537-7cb3-4523-b228-d4d28c57986c","Type":"ContainerStarted","Data":"88642ab73ab5a6b4645868da214715a8579fc3849d8bb4f3009c8dfe532dd514"} Nov 03 19:30:01.620939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:01.620728 4948 scope.go:122] "RemoveContainer" containerID="5f2202a048790fe05bdd4d6d98cfd65dca26ebc9f7964b90e09759c47d046d42" Nov 03 19:30:02.628664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:02.628585 4948 generic.go:356] "Generic (PLEG): container finished" podID="28424537-7cb3-4523-b228-d4d28c57986c" containerID="5b6416eaf2e769523be8f84ed2b8bafaf05c06dd2f073b5a7ae77a46470f05b4" exitCode=60 Nov 03 19:30:02.629089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:02.628668 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" event={"ID":"28424537-7cb3-4523-b228-d4d28c57986c","Type":"ContainerDied","Data":"5b6416eaf2e769523be8f84ed2b8bafaf05c06dd2f073b5a7ae77a46470f05b4"} Nov 03 19:30:02.629089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:02.628742 4948 scope.go:122] "RemoveContainer" containerID="5f2202a048790fe05bdd4d6d98cfd65dca26ebc9f7964b90e09759c47d046d42" Nov 03 19:30:02.629089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:02.629011 4948 scope.go:122] "RemoveContainer" containerID="5b6416eaf2e769523be8f84ed2b8bafaf05c06dd2f073b5a7ae77a46470f05b4" Nov 03 19:30:02.629314 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:30:02.629293 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895570-zl6hf_odh-ai-gateway-infra(28424537-7cb3-4523-b228-d4d28c57986c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" podUID="28424537-7cb3-4523-b228-d4d28c57986c" Nov 03 19:30:02.792568 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:30:02.792533 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:30:03.636625 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:03.636588 4948 scope.go:122] "RemoveContainer" containerID="5b6416eaf2e769523be8f84ed2b8bafaf05c06dd2f073b5a7ae77a46470f05b4" Nov 03 19:30:03.637033 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:30:03.636844 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895570-zl6hf_odh-ai-gateway-infra(28424537-7cb3-4523-b228-d4d28c57986c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" podUID="28424537-7cb3-4523-b228-d4d28c57986c" Nov 03 19:30:16.746003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:16.745973 4948 scope.go:122] "RemoveContainer" containerID="5b6416eaf2e769523be8f84ed2b8bafaf05c06dd2f073b5a7ae77a46470f05b4" Nov 03 19:30:17.726680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:17.726646 4948 generic.go:356] "Generic (PLEG): container finished" podID="28424537-7cb3-4523-b228-d4d28c57986c" containerID="6ee0780d43b831edb5b496e986f92525c4769fb1819822a73b8b25016b6182cb" exitCode=60 Nov 03 19:30:17.726861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:17.726737 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" event={"ID":"28424537-7cb3-4523-b228-d4d28c57986c","Type":"ContainerDied","Data":"6ee0780d43b831edb5b496e986f92525c4769fb1819822a73b8b25016b6182cb"} Nov 03 19:30:17.726861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:17.726804 4948 scope.go:122] "RemoveContainer" containerID="5b6416eaf2e769523be8f84ed2b8bafaf05c06dd2f073b5a7ae77a46470f05b4" Nov 03 19:30:17.727145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:17.727127 4948 scope.go:122] "RemoveContainer" containerID="6ee0780d43b831edb5b496e986f92525c4769fb1819822a73b8b25016b6182cb" Nov 03 19:30:17.727404 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:30:17.727380 4948 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895570-zl6hf_odh-ai-gateway-infra(28424537-7cb3-4523-b228-d4d28c57986c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" podUID="28424537-7cb3-4523-b228-d4d28c57986c" Nov 03 19:30:18.855196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:18.855157 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf"] Nov 03 19:30:18.998423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:18.998397 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:19.093038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.093011 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28424537-7cb3-4523-b228-d4d28c57986c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28424537-7cb3-4523-b228-d4d28c57986c-maas-api-service-ca\") pod \"28424537-7cb3-4523-b228-d4d28c57986c\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " Nov 03 19:30:19.093192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.093053 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28424537-7cb3-4523-b228-d4d28c57986c-kube-api-access-zlgzm\" (UniqueName: \"kubernetes.io/projected/28424537-7cb3-4523-b228-d4d28c57986c-kube-api-access-zlgzm\") pod \"28424537-7cb3-4523-b228-d4d28c57986c\" (UID: \"28424537-7cb3-4523-b228-d4d28c57986c\") " Nov 03 19:30:19.093364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.093336 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28424537-7cb3-4523-b228-d4d28c57986c-maas-api-service-ca" pod "28424537-7cb3-4523-b228-d4d28c57986c" (UID: "28424537-7cb3-4523-b228-d4d28c57986c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:19.096464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.096442 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28424537-7cb3-4523-b228-d4d28c57986c-kube-api-access-zlgzm" pod "28424537-7cb3-4523-b228-d4d28c57986c" (UID: "28424537-7cb3-4523-b228-d4d28c57986c"). InnerVolumeSpecName "kube-api-access-zlgzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:19.194334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.194303 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28424537-7cb3-4523-b228-d4d28c57986c-maas-api-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.194334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.194335 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgzm\" (UniqueName: \"kubernetes.io/projected/28424537-7cb3-4523-b228-d4d28c57986c-kube-api-access-zlgzm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.741967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.741936 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" Nov 03 19:30:19.741967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.741952 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf" event={"ID":"28424537-7cb3-4523-b228-d4d28c57986c","Type":"ContainerDied","Data":"88642ab73ab5a6b4645868da214715a8579fc3849d8bb4f3009c8dfe532dd514"} Nov 03 19:30:19.742252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.742007 4948 scope.go:122] "RemoveContainer" containerID="6ee0780d43b831edb5b496e986f92525c4769fb1819822a73b8b25016b6182cb" Nov 03 19:30:19.910136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.910107 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf"] Nov 03 19:30:19.961086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:19.961046 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-zl6hf"] Nov 03 19:30:20.754511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:20.754479 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28424537-7cb3-4523-b228-d4d28c57986c" path="/var/lib/kubelet/pods/28424537-7cb3-4523-b228-d4d28c57986c/volumes" Nov 03 19:30:26.043919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.043884 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb"] Nov 03 19:30:26.044322 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.044309 4948 cpu_manager.go:414] "containerMap: removing container" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:30:26.044368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.044323 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:30:26.044368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.044337 4948 cpu_manager.go:414] "containerMap: removing container" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:30:26.044368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.044342 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:30:26.044475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.044426 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:30:26.044475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.044435 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:30:26.044475 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.044444 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:30:26.047835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.047817 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.104925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.104886 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb"] Nov 03 19:30:26.152598 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.152564 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:30:26.152782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.152624 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:30:26.152782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.152661 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:30:26.152782 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.152731 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:30:26.162117 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.162088 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ac79a-b90c-445e-b357-107063b75271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.162246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.162124 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtsk\" (UniqueName: \"kubernetes.io/projected/c75ac79a-b90c-445e-b357-107063b75271-kube-api-access-hdtsk\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.162246 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.162203 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ac79a-b90c-445e-b357-107063b75271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.263655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.263618 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ac79a-b90c-445e-b357-107063b75271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.263655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.263660 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtsk\" (UniqueName: \"kubernetes.io/projected/c75ac79a-b90c-445e-b357-107063b75271-kube-api-access-hdtsk\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.263913 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.263723 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ac79a-b90c-445e-b357-107063b75271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.264412 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.264369 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ac79a-b90c-445e-b357-107063b75271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.266155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.266134 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ac79a-b90c-445e-b357-107063b75271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.326003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.325942 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtsk\" (UniqueName: \"kubernetes.io/projected/c75ac79a-b90c-445e-b357-107063b75271-kube-api-access-hdtsk\") pod \"maas-api-revoke-sub-d3fba056-bfzzb\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.361364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.361328 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:26.741182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.741150 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb"] Nov 03 19:30:26.742570 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:30:26.742538 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75ac79a_b90c_445e_b357_107063b75271.slice/crio-9b272f8639a9ce2e3d4ed324f33bb1ee97af129bf676f7b6cdad1e8a748bd8dc WatchSource:0}: Error finding container 9b272f8639a9ce2e3d4ed324f33bb1ee97af129bf676f7b6cdad1e8a748bd8dc: Status 404 returned error can't find the container with id 9b272f8639a9ce2e3d4ed324f33bb1ee97af129bf676f7b6cdad1e8a748bd8dc Nov 03 19:30:26.800075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:26.800039 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" event={"ID":"c75ac79a-b90c-445e-b357-107063b75271","Type":"ContainerStarted","Data":"9b272f8639a9ce2e3d4ed324f33bb1ee97af129bf676f7b6cdad1e8a748bd8dc"} Nov 03 19:30:27.809576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:27.809538 4948 generic.go:356] "Generic (PLEG): container finished" podID="c75ac79a-b90c-445e-b357-107063b75271" containerID="a51b2525aea82e2e82a38b7c1d209529c41f36d8340eaa2af4712c2eb8ae3d15" exitCode=0 Nov 03 19:30:27.810051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:27.809610 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" event={"ID":"c75ac79a-b90c-445e-b357-107063b75271","Type":"ContainerDied","Data":"a51b2525aea82e2e82a38b7c1d209529c41f36d8340eaa2af4712c2eb8ae3d15"} Nov 03 19:30:29.014413 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.014380 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:29.092675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.092588 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75ac79a-b90c-445e-b357-107063b75271-kube-api-access-hdtsk\" (UniqueName: \"kubernetes.io/projected/c75ac79a-b90c-445e-b357-107063b75271-kube-api-access-hdtsk\") pod \"c75ac79a-b90c-445e-b357-107063b75271\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " Nov 03 19:30:29.092978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.092956 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75ac79a-b90c-445e-b357-107063b75271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ac79a-b90c-445e-b357-107063b75271-maas-api-xks-ca\") pod \"c75ac79a-b90c-445e-b357-107063b75271\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " Nov 03 19:30:29.093092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.093064 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75ac79a-b90c-445e-b357-107063b75271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ac79a-b90c-445e-b357-107063b75271-maas-api-openshift-ca\") pod \"c75ac79a-b90c-445e-b357-107063b75271\" (UID: \"c75ac79a-b90c-445e-b357-107063b75271\") " Nov 03 19:30:29.093577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.093551 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75ac79a-b90c-445e-b357-107063b75271-maas-api-openshift-ca" pod "c75ac79a-b90c-445e-b357-107063b75271" (UID: "c75ac79a-b90c-445e-b357-107063b75271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:29.095504 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.095473 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75ac79a-b90c-445e-b357-107063b75271-kube-api-access-hdtsk" pod "c75ac79a-b90c-445e-b357-107063b75271" (UID: "c75ac79a-b90c-445e-b357-107063b75271"). InnerVolumeSpecName "kube-api-access-hdtsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:29.095800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.095761 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75ac79a-b90c-445e-b357-107063b75271-maas-api-xks-ca" pod "c75ac79a-b90c-445e-b357-107063b75271" (UID: "c75ac79a-b90c-445e-b357-107063b75271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:30:29.194781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.194718 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ac79a-b90c-445e-b357-107063b75271-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.194781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.194783 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdtsk\" (UniqueName: \"kubernetes.io/projected/c75ac79a-b90c-445e-b357-107063b75271-kube-api-access-hdtsk\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.195060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.194802 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ac79a-b90c-445e-b357-107063b75271-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:29.893733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.890018 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" event={"ID":"c75ac79a-b90c-445e-b357-107063b75271","Type":"ContainerDied","Data":"9b272f8639a9ce2e3d4ed324f33bb1ee97af129bf676f7b6cdad1e8a748bd8dc"} Nov 03 19:30:29.893733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.890087 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b272f8639a9ce2e3d4ed324f33bb1ee97af129bf676f7b6cdad1e8a748bd8dc" Nov 03 19:30:29.893733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:29.890164 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb" Nov 03 19:30:32.793282 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:30:32.793238 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:30:45.134051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:45.134003 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp"] Nov 03 19:30:45.139557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:45.139465 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c0cda-ssctp"] Nov 03 19:30:46.770115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:30:46.770075 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb6bcdf-a951-49cb-ae86-180449f264c7" path="/var/lib/kubelet/pods/5eb6bcdf-a951-49cb-ae86-180449f264c7/volumes" Nov 03 19:31:00.758753 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.758653 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6"] Nov 03 19:31:00.759306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.759285 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c75ac79a-b90c-445e-b357-107063b75271" containerName="revoke-keys" Nov 03 19:31:00.759395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.759309 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75ac79a-b90c-445e-b357-107063b75271" containerName="revoke-keys" Nov 03 19:31:00.759395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.759331 4948 cpu_manager.go:414] "containerMap: removing container" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:31:00.759395 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.759339 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28424537-7cb3-4523-b228-d4d28c57986c" containerName="cleanup" Nov 03 19:31:00.759567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.759449 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75ac79a-b90c-445e-b357-107063b75271" containerName="revoke-keys" Nov 03 19:31:00.762677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.762658 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.816902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.816864 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6"] Nov 03 19:31:00.826671 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.826639 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.826871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.826694 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j78t\" (UniqueName: \"kubernetes.io/projected/c42cbe7f-5825-435d-89e9-7398b7ee9e40-kube-api-access-7j78t\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.826969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.826904 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.864685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.864651 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:31:00.864895 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.864879 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:31:00.864980 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.864956 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:31:00.866947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.866926 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:31:00.931432 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.931395 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.931931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.931676 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j78t\" (UniqueName: \"kubernetes.io/projected/c42cbe7f-5825-435d-89e9-7398b7ee9e40-kube-api-access-7j78t\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.931931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.931792 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.932641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.932614 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.939215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.939190 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:00.994667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:00.994634 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j78t\" (UniqueName: \"kubernetes.io/projected/c42cbe7f-5825-435d-89e9-7398b7ee9e40-kube-api-access-7j78t\") pod \"maas-api-revoke-sub-ae594979-8rqq6\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:01.075438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:01.075360 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:01.455827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:01.455794 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6"] Nov 03 19:31:01.456756 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:31:01.456720 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42cbe7f_5825_435d_89e9_7398b7ee9e40.slice/crio-4e87508a18681914b46c6d9602c5ffb96248c27afb62aec0073d52c57414af62 WatchSource:0}: Error finding container 4e87508a18681914b46c6d9602c5ffb96248c27afb62aec0073d52c57414af62: Status 404 returned error can't find the container with id 4e87508a18681914b46c6d9602c5ffb96248c27afb62aec0073d52c57414af62 Nov 03 19:31:02.147635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:02.147543 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" event={"ID":"c42cbe7f-5825-435d-89e9-7398b7ee9e40","Type":"ContainerStarted","Data":"583338ce681bb2635783b1e6e8078920e1a0b0bc6bd8a3a24e898ab4362465e7"} Nov 03 19:31:02.147635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:02.147603 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" event={"ID":"c42cbe7f-5825-435d-89e9-7398b7ee9e40","Type":"ContainerStarted","Data":"4e87508a18681914b46c6d9602c5ffb96248c27afb62aec0073d52c57414af62"} Nov 03 19:31:02.265734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:02.265662 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" podStartSLOduration=2.265644896 podStartE2EDuration="2.265644896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:02.265234083 +0000 UTC m=+3191.748230000" watchObservedRunningTime="2026-11-03 19:31:02.265644896 +0000 UTC m=+3191.748640809" Nov 03 19:31:02.794069 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:31:02.794033 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:31:03.155005 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:03.154913 4948 generic.go:356] "Generic (PLEG): container finished" podID="c42cbe7f-5825-435d-89e9-7398b7ee9e40" containerID="583338ce681bb2635783b1e6e8078920e1a0b0bc6bd8a3a24e898ab4362465e7" exitCode=0 Nov 03 19:31:03.155436 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:03.154997 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" event={"ID":"c42cbe7f-5825-435d-89e9-7398b7ee9e40","Type":"ContainerDied","Data":"583338ce681bb2635783b1e6e8078920e1a0b0bc6bd8a3a24e898ab4362465e7"} Nov 03 19:31:04.301067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.301034 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:04.365109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.365071 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42cbe7f-5825-435d-89e9-7398b7ee9e40-kube-api-access-7j78t\" (UniqueName: \"kubernetes.io/projected/c42cbe7f-5825-435d-89e9-7398b7ee9e40-kube-api-access-7j78t\") pod \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " Nov 03 19:31:04.365275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.365154 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-xks-ca\") pod \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " Nov 03 19:31:04.365275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.365206 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-openshift-ca\") pod \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\" (UID: \"c42cbe7f-5825-435d-89e9-7398b7ee9e40\") " Nov 03 19:31:04.365610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.365585 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-openshift-ca" pod "c42cbe7f-5825-435d-89e9-7398b7ee9e40" (UID: "c42cbe7f-5825-435d-89e9-7398b7ee9e40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:04.367326 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.367293 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42cbe7f-5825-435d-89e9-7398b7ee9e40-kube-api-access-7j78t" pod "c42cbe7f-5825-435d-89e9-7398b7ee9e40" (UID: "c42cbe7f-5825-435d-89e9-7398b7ee9e40"). InnerVolumeSpecName "kube-api-access-7j78t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:04.367443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.367344 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-xks-ca" pod "c42cbe7f-5825-435d-89e9-7398b7ee9e40" (UID: "c42cbe7f-5825-435d-89e9-7398b7ee9e40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:31:04.466291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.466246 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j78t\" (UniqueName: \"kubernetes.io/projected/c42cbe7f-5825-435d-89e9-7398b7ee9e40-kube-api-access-7j78t\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:04.466291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.466287 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:04.466515 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:04.466301 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42cbe7f-5825-435d-89e9-7398b7ee9e40-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:05.168572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:05.168476 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" Nov 03 19:31:05.168572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:05.168511 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6" event={"ID":"c42cbe7f-5825-435d-89e9-7398b7ee9e40","Type":"ContainerDied","Data":"4e87508a18681914b46c6d9602c5ffb96248c27afb62aec0073d52c57414af62"} Nov 03 19:31:05.168572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:05.168559 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e87508a18681914b46c6d9602c5ffb96248c27afb62aec0073d52c57414af62" Nov 03 19:31:21.663039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:21.662993 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fmq4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:31:21.663833 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:21.663795 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fmq4c" podUID="ab5a14d7-8474-45ff-ae36-87c0aabad985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:31:31.874942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.874905 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv"] Nov 03 19:31:31.875491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.875345 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c42cbe7f-5825-435d-89e9-7398b7ee9e40" containerName="revoke-keys" Nov 03 19:31:31.875491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.875358 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42cbe7f-5825-435d-89e9-7398b7ee9e40" containerName="revoke-keys" Nov 03 19:31:31.875491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.875446 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42cbe7f-5825-435d-89e9-7398b7ee9e40" containerName="revoke-keys" Nov 03 19:31:31.878684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.878659 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:31.933341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.933305 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv"] Nov 03 19:31:31.978662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.978630 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkrq\" (UniqueName: \"kubernetes.io/projected/ac82fbbf-ef97-425e-a948-c20cc46a4dae-kube-api-access-6gkrq\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:31.978841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.978676 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:31.978841 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.978796 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:31.980675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.980648 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:31:31.980813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.980672 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:31:31.980813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.980721 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:31:31.982977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:31.982959 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:31:32.079547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.079515 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:32.079728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.079594 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkrq\" (UniqueName: \"kubernetes.io/projected/ac82fbbf-ef97-425e-a948-c20cc46a4dae-kube-api-access-6gkrq\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:32.079728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.079647 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:32.080241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.080218 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:32.081932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.081909 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:32.141659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.141604 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkrq\" (UniqueName: \"kubernetes.io/projected/ac82fbbf-ef97-425e-a948-c20cc46a4dae-kube-api-access-6gkrq\") pod \"maas-api-revoke-sub-a4aef7d7-mtxzv\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:32.192033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.192007 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:32.566734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:32.566709 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv"] Nov 03 19:31:32.567851 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:31:32.567810 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac82fbbf_ef97_425e_a948_c20cc46a4dae.slice/crio-bee21001d26a547a5075df9fb7afea7dab3750f25d59b85aeedefc7bea6c79fd WatchSource:0}: Error finding container bee21001d26a547a5075df9fb7afea7dab3750f25d59b85aeedefc7bea6c79fd: Status 404 returned error can't find the container with id bee21001d26a547a5075df9fb7afea7dab3750f25d59b85aeedefc7bea6c79fd Nov 03 19:31:32.792750 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:31:32.792689 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:31:33.434274 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:33.434240 4948 generic.go:356] "Generic (PLEG): container finished" podID="ac82fbbf-ef97-425e-a948-c20cc46a4dae" containerID="fd3af3d581c9b1eab2836f4231216123dede835ff43ecf742d58e5cb7d73caa6" exitCode=0 Nov 03 19:31:33.434627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:33.434326 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" event={"ID":"ac82fbbf-ef97-425e-a948-c20cc46a4dae","Type":"ContainerDied","Data":"fd3af3d581c9b1eab2836f4231216123dede835ff43ecf742d58e5cb7d73caa6"} Nov 03 19:31:33.434627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:33.434413 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" event={"ID":"ac82fbbf-ef97-425e-a948-c20cc46a4dae","Type":"ContainerStarted","Data":"bee21001d26a547a5075df9fb7afea7dab3750f25d59b85aeedefc7bea6c79fd"} Nov 03 19:31:34.577692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.577666 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:34.614151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.614109 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-openshift-ca\") pod \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " Nov 03 19:31:34.614151 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.614161 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac82fbbf-ef97-425e-a948-c20cc46a4dae-kube-api-access-6gkrq\" (UniqueName: \"kubernetes.io/projected/ac82fbbf-ef97-425e-a948-c20cc46a4dae-kube-api-access-6gkrq\") pod \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " Nov 03 19:31:34.614407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.614244 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-xks-ca\") pod \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\" (UID: \"ac82fbbf-ef97-425e-a948-c20cc46a4dae\") " Nov 03 19:31:34.614499 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.614474 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-openshift-ca" pod "ac82fbbf-ef97-425e-a948-c20cc46a4dae" (UID: "ac82fbbf-ef97-425e-a948-c20cc46a4dae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:31:34.616345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.616312 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-xks-ca" pod "ac82fbbf-ef97-425e-a948-c20cc46a4dae" (UID: "ac82fbbf-ef97-425e-a948-c20cc46a4dae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:31:34.616455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.616374 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac82fbbf-ef97-425e-a948-c20cc46a4dae-kube-api-access-6gkrq" pod "ac82fbbf-ef97-425e-a948-c20cc46a4dae" (UID: "ac82fbbf-ef97-425e-a948-c20cc46a4dae"). InnerVolumeSpecName "kube-api-access-6gkrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:31:34.716537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.716449 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:34.716537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.716484 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkrq\" (UniqueName: \"kubernetes.io/projected/ac82fbbf-ef97-425e-a948-c20cc46a4dae-kube-api-access-6gkrq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:34.716537 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:34.716499 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac82fbbf-ef97-425e-a948-c20cc46a4dae-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:31:35.447282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:35.447250 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" Nov 03 19:31:35.447470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:35.447285 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv" event={"ID":"ac82fbbf-ef97-425e-a948-c20cc46a4dae","Type":"ContainerDied","Data":"bee21001d26a547a5075df9fb7afea7dab3750f25d59b85aeedefc7bea6c79fd"} Nov 03 19:31:35.447470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:35.447333 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bee21001d26a547a5075df9fb7afea7dab3750f25d59b85aeedefc7bea6c79fd" Nov 03 19:31:54.065724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:54.065451 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c"] Nov 03 19:31:54.116294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:54.116257 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7e66ce-77z7c"] Nov 03 19:31:54.151818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:54.151783 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp"] Nov 03 19:31:54.153203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:54.153179 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde47b13-qndlp"] Nov 03 19:31:54.753487 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:54.753452 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7c4b28-314b-428b-b21a-34b890e5fa31" path="/var/lib/kubelet/pods/1d7c4b28-314b-428b-b21a-34b890e5fa31/volumes" Nov 03 19:31:54.753946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:31:54.753928 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b03e40-65b9-4472-9d70-d03451fed8ee" path="/var/lib/kubelet/pods/d6b03e40-65b9-4472-9d70-d03451fed8ee/volumes" Nov 03 19:32:00.939491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:00.939460 4948 scope.go:122] "RemoveContainer" containerID="a1ca6363febeff88476e20fe9d430678f3deb9c49e62a1716e1206cf333aa6e6" Nov 03 19:32:00.949109 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:00.949086 4948 scope.go:122] "RemoveContainer" containerID="6a11e94c5685446378a3deda7105d9fb3a6aee1a77966ddb146b8008752f61e1" Nov 03 19:32:00.957250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:00.957231 4948 scope.go:122] "RemoveContainer" containerID="8260466b47756877e5d0dd15c538ca64d8b451351a21da4dcee0c68beb09f2ef" Nov 03 19:32:02.562793 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.562759 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59"] Nov 03 19:32:02.563225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.563212 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ac82fbbf-ef97-425e-a948-c20cc46a4dae" containerName="revoke-keys" Nov 03 19:32:02.563270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.563226 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac82fbbf-ef97-425e-a948-c20cc46a4dae" containerName="revoke-keys" Nov 03 19:32:02.563330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.563321 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac82fbbf-ef97-425e-a948-c20cc46a4dae" containerName="revoke-keys" Nov 03 19:32:02.568017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.567995 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.622621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.622585 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59"] Nov 03 19:32:02.669453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.669420 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:32:02.669453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.669431 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:32:02.669669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.669468 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:32:02.672187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.672170 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:32:02.724203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.724170 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.724375 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.724240 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.724375 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.724312 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzbw\" (UniqueName: \"kubernetes.io/projected/16ca72c1-73e1-419f-89f0-06a762361f6f-kube-api-access-wxzbw\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.793597 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:32:02.793557 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:32:02.825238 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.825161 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.825402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.825344 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.825402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.825391 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzbw\" (UniqueName: \"kubernetes.io/projected/16ca72c1-73e1-419f-89f0-06a762361f6f-kube-api-access-wxzbw\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.825999 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.825978 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.827650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.827626 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:02.887518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:02.887482 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzbw\" (UniqueName: \"kubernetes.io/projected/16ca72c1-73e1-419f-89f0-06a762361f6f-kube-api-access-wxzbw\") pod \"maas-api-revoke-sub-79805ca9-zbs59\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:03.181621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:03.181534 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:03.559997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:03.559969 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59"] Nov 03 19:32:03.561388 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:32:03.561359 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ca72c1_73e1_419f_89f0_06a762361f6f.slice/crio-6b7369fbb9ccde93c73169089d83f5a80542163f0acb0ad4d7a405eb2326d327 WatchSource:0}: Error finding container 6b7369fbb9ccde93c73169089d83f5a80542163f0acb0ad4d7a405eb2326d327: Status 404 returned error can't find the container with id 6b7369fbb9ccde93c73169089d83f5a80542163f0acb0ad4d7a405eb2326d327 Nov 03 19:32:03.645135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:03.645100 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" event={"ID":"16ca72c1-73e1-419f-89f0-06a762361f6f","Type":"ContainerStarted","Data":"6b7369fbb9ccde93c73169089d83f5a80542163f0acb0ad4d7a405eb2326d327"} Nov 03 19:32:04.653015 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:04.652977 4948 generic.go:356] "Generic (PLEG): container finished" podID="16ca72c1-73e1-419f-89f0-06a762361f6f" containerID="18b69f7285d3e193cf83a5590db134969a0bc8dcec39a9953876bf7bd129b60e" exitCode=0 Nov 03 19:32:04.653451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:04.653068 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" event={"ID":"16ca72c1-73e1-419f-89f0-06a762361f6f","Type":"ContainerDied","Data":"18b69f7285d3e193cf83a5590db134969a0bc8dcec39a9953876bf7bd129b60e"} Nov 03 19:32:05.791476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:05.791452 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:05.955345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:05.955313 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-xks-ca\") pod \"16ca72c1-73e1-419f-89f0-06a762361f6f\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " Nov 03 19:32:05.955503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:05.955357 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ca72c1-73e1-419f-89f0-06a762361f6f-kube-api-access-wxzbw\" (UniqueName: \"kubernetes.io/projected/16ca72c1-73e1-419f-89f0-06a762361f6f-kube-api-access-wxzbw\") pod \"16ca72c1-73e1-419f-89f0-06a762361f6f\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " Nov 03 19:32:05.955503 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:05.955392 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-openshift-ca\") pod \"16ca72c1-73e1-419f-89f0-06a762361f6f\" (UID: \"16ca72c1-73e1-419f-89f0-06a762361f6f\") " Nov 03 19:32:05.955851 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:05.955823 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-openshift-ca" pod "16ca72c1-73e1-419f-89f0-06a762361f6f" (UID: "16ca72c1-73e1-419f-89f0-06a762361f6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:05.957342 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:05.957313 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-xks-ca" pod "16ca72c1-73e1-419f-89f0-06a762361f6f" (UID: "16ca72c1-73e1-419f-89f0-06a762361f6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:05.957482 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:05.957467 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ca72c1-73e1-419f-89f0-06a762361f6f-kube-api-access-wxzbw" pod "16ca72c1-73e1-419f-89f0-06a762361f6f" (UID: "16ca72c1-73e1-419f-89f0-06a762361f6f"). InnerVolumeSpecName "kube-api-access-wxzbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:06.057223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:06.057189 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:06.057223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:06.057219 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxzbw\" (UniqueName: \"kubernetes.io/projected/16ca72c1-73e1-419f-89f0-06a762361f6f-kube-api-access-wxzbw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:06.057223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:06.057229 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ca72c1-73e1-419f-89f0-06a762361f6f-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:06.667646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:06.667612 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" event={"ID":"16ca72c1-73e1-419f-89f0-06a762361f6f","Type":"ContainerDied","Data":"6b7369fbb9ccde93c73169089d83f5a80542163f0acb0ad4d7a405eb2326d327"} Nov 03 19:32:06.667646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:06.667648 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59" Nov 03 19:32:06.667857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:06.667666 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b7369fbb9ccde93c73169089d83f5a80542163f0acb0ad4d7a405eb2326d327" Nov 03 19:32:19.822921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:19.822866 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8pc6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:32:19.824048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:19.823600 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8pc6b" podUID="4f2486e1-e5ca-46ac-aea5-9101cc6eb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:32:20.112907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:20.112799 4948 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-vdlrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:32:20.112907 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:20.112877 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-vdlrm" podUID="99a8205c-e0cd-4eeb-828a-c78ec53fe35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:32:26.299196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.299154 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57"] Nov 03 19:32:26.299875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.299847 4948 cpu_manager.go:414] "containerMap: removing container" podUID="16ca72c1-73e1-419f-89f0-06a762361f6f" containerName="revoke-keys" Nov 03 19:32:26.299875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.299871 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ca72c1-73e1-419f-89f0-06a762361f6f" containerName="revoke-keys" Nov 03 19:32:26.300047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.299964 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ca72c1-73e1-419f-89f0-06a762361f6f" containerName="revoke-keys" Nov 03 19:32:26.303028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.303009 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.358672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.358631 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57"] Nov 03 19:32:26.381462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.381424 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.381462 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.381470 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.381760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.381624 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwgw\" (UniqueName: \"kubernetes.io/projected/4036196c-3a43-4fe6-8f86-e0eccbdd5843-kube-api-access-9pwgw\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.404887 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.404846 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:32:26.404887 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.404864 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:32:26.404887 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.404846 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:32:26.405147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.404894 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:32:26.482623 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.482588 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwgw\" (UniqueName: \"kubernetes.io/projected/4036196c-3a43-4fe6-8f86-e0eccbdd5843-kube-api-access-9pwgw\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.482871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.482662 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.482871 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.482690 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.483383 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.483354 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.485113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.485080 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.545366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.545331 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwgw\" (UniqueName: \"kubernetes.io/projected/4036196c-3a43-4fe6-8f86-e0eccbdd5843-kube-api-access-9pwgw\") pod \"maas-api-revoke-sub-0b7e8edb-9td57\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.616664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.616580 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:26.997958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:26.997923 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57"] Nov 03 19:32:26.998499 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:32:26.998467 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4036196c_3a43_4fe6_8f86_e0eccbdd5843.slice/crio-e14295f3d24ba2303bd3d16fb8fe6299efa716b40a2d884e1815fad2802df0b7 WatchSource:0}: Error finding container e14295f3d24ba2303bd3d16fb8fe6299efa716b40a2d884e1815fad2802df0b7: Status 404 returned error can't find the container with id e14295f3d24ba2303bd3d16fb8fe6299efa716b40a2d884e1815fad2802df0b7 Nov 03 19:32:27.846229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:27.846202 4948 generic.go:356] "Generic (PLEG): container finished" podID="4036196c-3a43-4fe6-8f86-e0eccbdd5843" containerID="b41b7d4f8a47bcd72b57de5141c0cd0aed0bfe79a969fc0e300844805c06ed93" exitCode=0 Nov 03 19:32:27.846592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:27.846288 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" event={"ID":"4036196c-3a43-4fe6-8f86-e0eccbdd5843","Type":"ContainerDied","Data":"b41b7d4f8a47bcd72b57de5141c0cd0aed0bfe79a969fc0e300844805c06ed93"} Nov 03 19:32:27.846592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:27.846345 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" event={"ID":"4036196c-3a43-4fe6-8f86-e0eccbdd5843","Type":"ContainerStarted","Data":"e14295f3d24ba2303bd3d16fb8fe6299efa716b40a2d884e1815fad2802df0b7"} Nov 03 19:32:29.000717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.000676 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:29.107524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.107491 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4036196c-3a43-4fe6-8f86-e0eccbdd5843-kube-api-access-9pwgw\" (UniqueName: \"kubernetes.io/projected/4036196c-3a43-4fe6-8f86-e0eccbdd5843-kube-api-access-9pwgw\") pod \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " Nov 03 19:32:29.107763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.107559 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-xks-ca\") pod \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " Nov 03 19:32:29.107763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.107600 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-openshift-ca\") pod \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\" (UID: \"4036196c-3a43-4fe6-8f86-e0eccbdd5843\") " Nov 03 19:32:29.108076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.108043 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-openshift-ca" pod "4036196c-3a43-4fe6-8f86-e0eccbdd5843" (UID: "4036196c-3a43-4fe6-8f86-e0eccbdd5843"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:29.109750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.109726 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-xks-ca" pod "4036196c-3a43-4fe6-8f86-e0eccbdd5843" (UID: "4036196c-3a43-4fe6-8f86-e0eccbdd5843"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:29.109838 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.109764 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4036196c-3a43-4fe6-8f86-e0eccbdd5843-kube-api-access-9pwgw" pod "4036196c-3a43-4fe6-8f86-e0eccbdd5843" (UID: "4036196c-3a43-4fe6-8f86-e0eccbdd5843"). InnerVolumeSpecName "kube-api-access-9pwgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:29.208584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.208546 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwgw\" (UniqueName: \"kubernetes.io/projected/4036196c-3a43-4fe6-8f86-e0eccbdd5843-kube-api-access-9pwgw\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:29.208584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.208580 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:29.208584 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.208591 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036196c-3a43-4fe6-8f86-e0eccbdd5843-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:29.871898 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.871867 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" Nov 03 19:32:29.872095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.871902 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57" event={"ID":"4036196c-3a43-4fe6-8f86-e0eccbdd5843","Type":"ContainerDied","Data":"e14295f3d24ba2303bd3d16fb8fe6299efa716b40a2d884e1815fad2802df0b7"} Nov 03 19:32:29.872095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:29.871951 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14295f3d24ba2303bd3d16fb8fe6299efa716b40a2d884e1815fad2802df0b7" Nov 03 19:32:32.793926 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:32:32.793838 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:32:37.619911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.619875 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd"] Nov 03 19:32:37.620370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.620318 4948 cpu_manager.go:414] "containerMap: removing container" podUID="4036196c-3a43-4fe6-8f86-e0eccbdd5843" containerName="revoke-keys" Nov 03 19:32:37.620370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.620331 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4036196c-3a43-4fe6-8f86-e0eccbdd5843" containerName="revoke-keys" Nov 03 19:32:37.620449 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.620439 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="4036196c-3a43-4fe6-8f86-e0eccbdd5843" containerName="revoke-keys" Nov 03 19:32:37.623897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.623877 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.677067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.677031 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:32:37.677250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.677175 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:32:37.677250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.677180 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:32:37.705795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.705756 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.705795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.705800 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnr85\" (UniqueName: \"kubernetes.io/projected/9dc0635c-3dda-482f-8701-9c5bac2e73a1-kube-api-access-bnr85\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.706043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.705883 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.728178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.728143 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:32:37.731229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.731200 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd"] Nov 03 19:32:37.806666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.806622 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.806878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.806819 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.806878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.806839 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnr85\" (UniqueName: \"kubernetes.io/projected/9dc0635c-3dda-482f-8701-9c5bac2e73a1-kube-api-access-bnr85\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.807407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.807370 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.809341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.809319 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.868578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.868544 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnr85\" (UniqueName: \"kubernetes.io/projected/9dc0635c-3dda-482f-8701-9c5bac2e73a1-kube-api-access-bnr85\") pod \"maas-api-revoke-sub-3d4156f1-jnxcd\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:37.937650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:37.937619 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:38.314641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:38.314607 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd"] Nov 03 19:32:38.314807 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:32:38.314684 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc0635c_3dda_482f_8701_9c5bac2e73a1.slice/crio-944ab06f59b67e05a7462281242b9cd98d8df669b58adab7ec7e985dea3b7b57 WatchSource:0}: Error finding container 944ab06f59b67e05a7462281242b9cd98d8df669b58adab7ec7e985dea3b7b57: Status 404 returned error can't find the container with id 944ab06f59b67e05a7462281242b9cd98d8df669b58adab7ec7e985dea3b7b57 Nov 03 19:32:38.928545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:38.928449 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" event={"ID":"9dc0635c-3dda-482f-8701-9c5bac2e73a1","Type":"ContainerStarted","Data":"f79f7adf75a0b430a58c700374cad7e0f3adfa4affac5bdf21591f66217941fa"} Nov 03 19:32:38.928545 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:38.928513 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" event={"ID":"9dc0635c-3dda-482f-8701-9c5bac2e73a1","Type":"ContainerStarted","Data":"944ab06f59b67e05a7462281242b9cd98d8df669b58adab7ec7e985dea3b7b57"} Nov 03 19:32:39.046414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:39.046356 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" podStartSLOduration=2.046341017 podStartE2EDuration="2.046341017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:39.045865885 +0000 UTC m=+3288.528861799" watchObservedRunningTime="2026-11-03 19:32:39.046341017 +0000 UTC m=+3288.529336931" Nov 03 19:32:39.955874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:39.955202 4948 generic.go:356] "Generic (PLEG): container finished" podID="9dc0635c-3dda-482f-8701-9c5bac2e73a1" containerID="f79f7adf75a0b430a58c700374cad7e0f3adfa4affac5bdf21591f66217941fa" exitCode=0 Nov 03 19:32:39.955874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:39.955439 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" event={"ID":"9dc0635c-3dda-482f-8701-9c5bac2e73a1","Type":"ContainerDied","Data":"f79f7adf75a0b430a58c700374cad7e0f3adfa4affac5bdf21591f66217941fa"} Nov 03 19:32:41.585486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.585455 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:41.702747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.702286 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-xks-ca\") pod \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " Nov 03 19:32:41.702747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.702368 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc0635c-3dda-482f-8701-9c5bac2e73a1-kube-api-access-bnr85\" (UniqueName: \"kubernetes.io/projected/9dc0635c-3dda-482f-8701-9c5bac2e73a1-kube-api-access-bnr85\") pod \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " Nov 03 19:32:41.702747 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.702412 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-openshift-ca\") pod \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\" (UID: \"9dc0635c-3dda-482f-8701-9c5bac2e73a1\") " Nov 03 19:32:41.703079 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.702814 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-openshift-ca" pod "9dc0635c-3dda-482f-8701-9c5bac2e73a1" (UID: "9dc0635c-3dda-482f-8701-9c5bac2e73a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:41.705039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.704991 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc0635c-3dda-482f-8701-9c5bac2e73a1-kube-api-access-bnr85" pod "9dc0635c-3dda-482f-8701-9c5bac2e73a1" (UID: "9dc0635c-3dda-482f-8701-9c5bac2e73a1"). InnerVolumeSpecName "kube-api-access-bnr85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:41.706837 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.706803 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-xks-ca" pod "9dc0635c-3dda-482f-8701-9c5bac2e73a1" (UID: "9dc0635c-3dda-482f-8701-9c5bac2e73a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:41.803948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.803908 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:41.804133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.803963 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnr85\" (UniqueName: \"kubernetes.io/projected/9dc0635c-3dda-482f-8701-9c5bac2e73a1-kube-api-access-bnr85\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:41.804133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.803974 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc0635c-3dda-482f-8701-9c5bac2e73a1-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:41.988198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.988118 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" event={"ID":"9dc0635c-3dda-482f-8701-9c5bac2e73a1","Type":"ContainerDied","Data":"944ab06f59b67e05a7462281242b9cd98d8df669b58adab7ec7e985dea3b7b57"} Nov 03 19:32:41.988198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.988170 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944ab06f59b67e05a7462281242b9cd98d8df669b58adab7ec7e985dea3b7b57" Nov 03 19:32:41.988198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:41.988129 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd" Nov 03 19:32:48.050577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:48.050536 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x"] Nov 03 19:32:48.101346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:48.101307 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vcn2x"] Nov 03 19:32:48.758447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:48.758399 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15a590f-8e60-4871-b57a-09ed217f5c36" path="/var/lib/kubelet/pods/b15a590f-8e60-4871-b57a-09ed217f5c36/volumes" Nov 03 19:32:55.763984 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:55.763957 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:32:55.769957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:55.769933 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:32:56.025188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:56.025111 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:32:56.032251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:32:56.032227 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:33:00.993278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:00.993238 4948 scope.go:122] "RemoveContainer" containerID="52de5a73d36cacdc9a8a2e1244cd18e4fe8ef72660526e2be9b5f4683f2f3a14" Nov 03 19:33:01.046490 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:01.046462 4948 scope.go:122] "RemoveContainer" containerID="3339801d61b9fb360c34387a4b274d5b372b9fd29c9cbd7f1e066e399ae8e741" Nov 03 19:33:01.082431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:01.082403 4948 scope.go:122] "RemoveContainer" containerID="6551d71ece4c55962b1bed089d991115c0451a98b61461afdbfe8e5b4ca5d5f1" Nov 03 19:33:02.793801 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:33:02.793762 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:33:11.220810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.220770 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd"] Nov 03 19:33:11.271598 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.271563 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jnxcd"] Nov 03 19:33:11.336247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.336205 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq"] Nov 03 19:33:11.336682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.336669 4948 cpu_manager.go:414] "containerMap: removing container" podUID="9dc0635c-3dda-482f-8701-9c5bac2e73a1" containerName="revoke-keys" Nov 03 19:33:11.336777 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.336684 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc0635c-3dda-482f-8701-9c5bac2e73a1" containerName="revoke-keys" Nov 03 19:33:11.336828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.336789 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc0635c-3dda-482f-8701-9c5bac2e73a1" containerName="revoke-keys" Nov 03 19:33:11.342134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.342112 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.390277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.390241 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.390277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.390289 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2fp\" (UniqueName: \"kubernetes.io/projected/b7edfad3-8b08-47d6-92cc-52170b43e4b3-kube-api-access-ws2fp\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.390544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.390398 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.394141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.394106 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq"] Nov 03 19:33:11.395916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.395888 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:33:11.396111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.395889 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:33:11.396111 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.395910 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:33:11.396292 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.396004 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:33:11.491500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.491412 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.491500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.491455 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2fp\" (UniqueName: \"kubernetes.io/projected/b7edfad3-8b08-47d6-92cc-52170b43e4b3-kube-api-access-ws2fp\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.491500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.491485 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.492291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.492265 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.493819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.493799 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.554430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.554393 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2fp\" (UniqueName: \"kubernetes.io/projected/b7edfad3-8b08-47d6-92cc-52170b43e4b3-kube-api-access-ws2fp\") pod \"maas-api-revoke-sub-3d4156f1-5zbrq\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:11.655825 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:11.655789 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:12.033607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:12.033579 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq"] Nov 03 19:33:12.034653 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:33:12.034622 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7edfad3_8b08_47d6_92cc_52170b43e4b3.slice/crio-5fdccf34eccb29d3f8400e56bcfc3dfb65427962a81f11e7c2fc461a2429306a WatchSource:0}: Error finding container 5fdccf34eccb29d3f8400e56bcfc3dfb65427962a81f11e7c2fc461a2429306a: Status 404 returned error can't find the container with id 5fdccf34eccb29d3f8400e56bcfc3dfb65427962a81f11e7c2fc461a2429306a Nov 03 19:33:12.404522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:12.404484 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" event={"ID":"b7edfad3-8b08-47d6-92cc-52170b43e4b3","Type":"ContainerStarted","Data":"674bf4d15a7f77c6233b7ebd4a6114aefea8d28e8c391730f14284a3881894c9"} Nov 03 19:33:12.404957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:12.404542 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" event={"ID":"b7edfad3-8b08-47d6-92cc-52170b43e4b3","Type":"ContainerStarted","Data":"5fdccf34eccb29d3f8400e56bcfc3dfb65427962a81f11e7c2fc461a2429306a"} Nov 03 19:33:12.522663 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:12.522613 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" podStartSLOduration=1.522599328 podStartE2EDuration="1.522599328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:12.52217668 +0000 UTC m=+3322.005172606" watchObservedRunningTime="2026-11-03 19:33:12.522599328 +0000 UTC m=+3322.005595242" Nov 03 19:33:12.753089 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:12.753054 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc0635c-3dda-482f-8701-9c5bac2e73a1" path="/var/lib/kubelet/pods/9dc0635c-3dda-482f-8701-9c5bac2e73a1/volumes" Nov 03 19:33:13.412011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:13.411979 4948 generic.go:356] "Generic (PLEG): container finished" podID="b7edfad3-8b08-47d6-92cc-52170b43e4b3" containerID="674bf4d15a7f77c6233b7ebd4a6114aefea8d28e8c391730f14284a3881894c9" exitCode=0 Nov 03 19:33:13.412484 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:13.412064 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" event={"ID":"b7edfad3-8b08-47d6-92cc-52170b43e4b3","Type":"ContainerDied","Data":"674bf4d15a7f77c6233b7ebd4a6114aefea8d28e8c391730f14284a3881894c9"} Nov 03 19:33:14.557757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.557732 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:14.619465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.619428 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7edfad3-8b08-47d6-92cc-52170b43e4b3-kube-api-access-ws2fp\" (UniqueName: \"kubernetes.io/projected/b7edfad3-8b08-47d6-92cc-52170b43e4b3-kube-api-access-ws2fp\") pod \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " Nov 03 19:33:14.619646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.619495 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-openshift-ca\") pod \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " Nov 03 19:33:14.619646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.619519 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-xks-ca\") pod \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\" (UID: \"b7edfad3-8b08-47d6-92cc-52170b43e4b3\") " Nov 03 19:33:14.619931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.619905 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-openshift-ca" pod "b7edfad3-8b08-47d6-92cc-52170b43e4b3" (UID: "b7edfad3-8b08-47d6-92cc-52170b43e4b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:14.621708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.621671 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-xks-ca" pod "b7edfad3-8b08-47d6-92cc-52170b43e4b3" (UID: "b7edfad3-8b08-47d6-92cc-52170b43e4b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:14.621824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.621807 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7edfad3-8b08-47d6-92cc-52170b43e4b3-kube-api-access-ws2fp" pod "b7edfad3-8b08-47d6-92cc-52170b43e4b3" (UID: "b7edfad3-8b08-47d6-92cc-52170b43e4b3"). InnerVolumeSpecName "kube-api-access-ws2fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:14.720983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.720952 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2fp\" (UniqueName: \"kubernetes.io/projected/b7edfad3-8b08-47d6-92cc-52170b43e4b3-kube-api-access-ws2fp\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:14.720983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.720982 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:14.721198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:14.720995 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7edfad3-8b08-47d6-92cc-52170b43e4b3-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:15.425842 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:15.425809 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" Nov 03 19:33:15.426037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:15.425808 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq" event={"ID":"b7edfad3-8b08-47d6-92cc-52170b43e4b3","Type":"ContainerDied","Data":"5fdccf34eccb29d3f8400e56bcfc3dfb65427962a81f11e7c2fc461a2429306a"} Nov 03 19:33:15.426037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:15.425935 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fdccf34eccb29d3f8400e56bcfc3dfb65427962a81f11e7c2fc461a2429306a" Nov 03 19:33:32.792875 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:33:32.792836 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:33:57.050464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:57.050423 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz"] Nov 03 19:33:57.101324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:57.101283 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6zjz"] Nov 03 19:33:58.752288 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:33:58.752254 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d9b1cc-da85-47aa-baf2-47b9e9d7f78f" path="/var/lib/kubelet/pods/77d9b1cc-da85-47aa-baf2-47b9e9d7f78f/volumes" Nov 03 19:34:01.178745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.178646 4948 scope.go:122] "RemoveContainer" containerID="abe3ac23eeadb7ada771fc5a37181aa80016600078217c100d40cf0561f6b4cc" Nov 03 19:34:01.636692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.636655 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp"] Nov 03 19:34:01.637119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.637105 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b7edfad3-8b08-47d6-92cc-52170b43e4b3" containerName="revoke-keys" Nov 03 19:34:01.637173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.637122 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7edfad3-8b08-47d6-92cc-52170b43e4b3" containerName="revoke-keys" Nov 03 19:34:01.637210 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.637205 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7edfad3-8b08-47d6-92cc-52170b43e4b3" containerName="revoke-keys" Nov 03 19:34:01.641835 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.641814 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.696083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.696053 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp"] Nov 03 19:34:01.743242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.743210 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:34:01.743242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.743227 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:34:01.743242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.743212 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:34:01.746202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.746182 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:34:01.759677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.759641 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.759677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.759676 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.759942 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.759715 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj88t\" (UniqueName: \"kubernetes.io/projected/759975d3-885b-4904-896f-0e2f3867dac3-kube-api-access-fj88t\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.860925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.860888 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.861110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.860940 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.861110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.860967 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj88t\" (UniqueName: \"kubernetes.io/projected/759975d3-885b-4904-896f-0e2f3867dac3-kube-api-access-fj88t\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.861662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.861636 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.863403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.863378 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.925758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.925654 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj88t\" (UniqueName: \"kubernetes.io/projected/759975d3-885b-4904-896f-0e2f3867dac3-kube-api-access-fj88t\") pod \"maas-api-revoke-sub-e2fc9c5b-2tkhp\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:01.955610 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:01.955576 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:02.334927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:02.334896 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp"] Nov 03 19:34:02.336288 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:34:02.336255 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759975d3_885b_4904_896f_0e2f3867dac3.slice/crio-a109b5fb98b3735d19f4a498205d6feb8728bc95f5395e3b8d92ea20db987025 WatchSource:0}: Error finding container a109b5fb98b3735d19f4a498205d6feb8728bc95f5395e3b8d92ea20db987025: Status 404 returned error can't find the container with id a109b5fb98b3735d19f4a498205d6feb8728bc95f5395e3b8d92ea20db987025 Nov 03 19:34:02.764498 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:02.764464 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" event={"ID":"759975d3-885b-4904-896f-0e2f3867dac3","Type":"ContainerStarted","Data":"012afdda0725045f6648e2f9eb73b0e45f9451c6ccc21d6d0ebd6487975887bc"} Nov 03 19:34:02.764745 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:02.764519 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" event={"ID":"759975d3-885b-4904-896f-0e2f3867dac3","Type":"ContainerStarted","Data":"a109b5fb98b3735d19f4a498205d6feb8728bc95f5395e3b8d92ea20db987025"} Nov 03 19:34:02.793011 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:34:02.792975 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:34:02.889303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:02.889250 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" podStartSLOduration=1.8892341099999999 podStartE2EDuration="1.88923411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:02.888748033 +0000 UTC m=+3372.371743947" watchObservedRunningTime="2026-11-03 19:34:02.88923411 +0000 UTC m=+3372.372230027" Nov 03 19:34:03.774236 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:03.774193 4948 generic.go:356] "Generic (PLEG): container finished" podID="759975d3-885b-4904-896f-0e2f3867dac3" containerID="012afdda0725045f6648e2f9eb73b0e45f9451c6ccc21d6d0ebd6487975887bc" exitCode=0 Nov 03 19:34:03.774923 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:03.774269 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" event={"ID":"759975d3-885b-4904-896f-0e2f3867dac3","Type":"ContainerDied","Data":"012afdda0725045f6648e2f9eb73b0e45f9451c6ccc21d6d0ebd6487975887bc"} Nov 03 19:34:04.919131 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:04.919104 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:04.993195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:04.993166 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-xks-ca\") pod \"759975d3-885b-4904-896f-0e2f3867dac3\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " Nov 03 19:34:04.993368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:04.993218 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-openshift-ca\") pod \"759975d3-885b-4904-896f-0e2f3867dac3\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " Nov 03 19:34:04.993368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:04.993270 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759975d3-885b-4904-896f-0e2f3867dac3-kube-api-access-fj88t\" (UniqueName: \"kubernetes.io/projected/759975d3-885b-4904-896f-0e2f3867dac3-kube-api-access-fj88t\") pod \"759975d3-885b-4904-896f-0e2f3867dac3\" (UID: \"759975d3-885b-4904-896f-0e2f3867dac3\") " Nov 03 19:34:04.993574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:04.993546 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-openshift-ca" pod "759975d3-885b-4904-896f-0e2f3867dac3" (UID: "759975d3-885b-4904-896f-0e2f3867dac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:04.995167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:04.995144 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-xks-ca" pod "759975d3-885b-4904-896f-0e2f3867dac3" (UID: "759975d3-885b-4904-896f-0e2f3867dac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:04.995268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:04.995191 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759975d3-885b-4904-896f-0e2f3867dac3-kube-api-access-fj88t" pod "759975d3-885b-4904-896f-0e2f3867dac3" (UID: "759975d3-885b-4904-896f-0e2f3867dac3"). InnerVolumeSpecName "kube-api-access-fj88t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:05.094310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:05.094218 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:05.094310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:05.094262 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759975d3-885b-4904-896f-0e2f3867dac3-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:05.094310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:05.094274 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj88t\" (UniqueName: \"kubernetes.io/projected/759975d3-885b-4904-896f-0e2f3867dac3-kube-api-access-fj88t\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:05.787565 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:05.787537 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" Nov 03 19:34:05.787758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:05.787578 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp" event={"ID":"759975d3-885b-4904-896f-0e2f3867dac3","Type":"ContainerDied","Data":"a109b5fb98b3735d19f4a498205d6feb8728bc95f5395e3b8d92ea20db987025"} Nov 03 19:34:05.787758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:05.787622 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a109b5fb98b3735d19f4a498205d6feb8728bc95f5395e3b8d92ea20db987025" Nov 03 19:34:12.418218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.418184 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w"] Nov 03 19:34:12.418657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.418628 4948 cpu_manager.go:414] "containerMap: removing container" podUID="759975d3-885b-4904-896f-0e2f3867dac3" containerName="revoke-keys" Nov 03 19:34:12.418657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.418642 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759975d3-885b-4904-896f-0e2f3867dac3" containerName="revoke-keys" Nov 03 19:34:12.418767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.418740 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="759975d3-885b-4904-896f-0e2f3867dac3" containerName="revoke-keys" Nov 03 19:34:12.421976 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.421958 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.477895 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.477848 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w"] Nov 03 19:34:12.524773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.524740 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:34:12.524974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.524776 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:34:12.524974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.524777 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:34:12.524974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.524833 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:34:12.568596 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.568564 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.568790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.568613 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.568854 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.568784 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b7jj\" (UniqueName: \"kubernetes.io/projected/8933f286-39a0-465f-9b6e-301557aa4157-kube-api-access-5b7jj\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.670086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.670006 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b7jj\" (UniqueName: \"kubernetes.io/projected/8933f286-39a0-465f-9b6e-301557aa4157-kube-api-access-5b7jj\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.670086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.670052 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.670086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.670084 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.670771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.670747 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.672410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.672385 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.732476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.732446 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b7jj\" (UniqueName: \"kubernetes.io/projected/8933f286-39a0-465f-9b6e-301557aa4157-kube-api-access-5b7jj\") pod \"maas-api-revoke-sub-5a29efb3-pqh7w\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:12.737389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:12.737372 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:13.117140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:13.117111 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w"] Nov 03 19:34:13.117962 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:34:13.117936 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8933f286_39a0_465f_9b6e_301557aa4157.slice/crio-03ffb9bd2a43b24f107c3b7422d25933897669226107549d20b5cb8d0a3580c5 WatchSource:0}: Error finding container 03ffb9bd2a43b24f107c3b7422d25933897669226107549d20b5cb8d0a3580c5: Status 404 returned error can't find the container with id 03ffb9bd2a43b24f107c3b7422d25933897669226107549d20b5cb8d0a3580c5 Nov 03 19:34:13.837074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:13.837035 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" event={"ID":"8933f286-39a0-465f-9b6e-301557aa4157","Type":"ContainerStarted","Data":"9387955ad4979db1b4350bd549d5e85f1e07a7d51dc774993a3e32c9223f3263"} Nov 03 19:34:13.837863 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:13.837102 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" event={"ID":"8933f286-39a0-465f-9b6e-301557aa4157","Type":"ContainerStarted","Data":"03ffb9bd2a43b24f107c3b7422d25933897669226107549d20b5cb8d0a3580c5"} Nov 03 19:34:13.955042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:13.954979 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" podStartSLOduration=1.954960302 podStartE2EDuration="1.954960302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:13.954758564 +0000 UTC m=+3383.437754477" watchObservedRunningTime="2026-11-03 19:34:13.954960302 +0000 UTC m=+3383.437956213" Nov 03 19:34:14.863742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:14.863682 4948 generic.go:356] "Generic (PLEG): container finished" podID="8933f286-39a0-465f-9b6e-301557aa4157" containerID="9387955ad4979db1b4350bd549d5e85f1e07a7d51dc774993a3e32c9223f3263" exitCode=0 Nov 03 19:34:14.864373 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:14.864347 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" event={"ID":"8933f286-39a0-465f-9b6e-301557aa4157","Type":"ContainerDied","Data":"9387955ad4979db1b4350bd549d5e85f1e07a7d51dc774993a3e32c9223f3263"} Nov 03 19:34:16.848072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.847932 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:16.864731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.863505 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-openshift-ca\") pod \"8933f286-39a0-465f-9b6e-301557aa4157\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " Nov 03 19:34:16.864731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.863588 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-xks-ca\") pod \"8933f286-39a0-465f-9b6e-301557aa4157\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " Nov 03 19:34:16.864731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.863746 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8933f286-39a0-465f-9b6e-301557aa4157-kube-api-access-5b7jj\" (UniqueName: \"kubernetes.io/projected/8933f286-39a0-465f-9b6e-301557aa4157-kube-api-access-5b7jj\") pod \"8933f286-39a0-465f-9b6e-301557aa4157\" (UID: \"8933f286-39a0-465f-9b6e-301557aa4157\") " Nov 03 19:34:16.867915 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.867882 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-openshift-ca" pod "8933f286-39a0-465f-9b6e-301557aa4157" (UID: "8933f286-39a0-465f-9b6e-301557aa4157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:16.881790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.881750 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-xks-ca" pod "8933f286-39a0-465f-9b6e-301557aa4157" (UID: "8933f286-39a0-465f-9b6e-301557aa4157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:16.907992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.907958 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8933f286-39a0-465f-9b6e-301557aa4157-kube-api-access-5b7jj" pod "8933f286-39a0-465f-9b6e-301557aa4157" (UID: "8933f286-39a0-465f-9b6e-301557aa4157"). InnerVolumeSpecName "kube-api-access-5b7jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:16.954780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.945060 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" event={"ID":"8933f286-39a0-465f-9b6e-301557aa4157","Type":"ContainerDied","Data":"03ffb9bd2a43b24f107c3b7422d25933897669226107549d20b5cb8d0a3580c5"} Nov 03 19:34:16.954780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.945121 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ffb9bd2a43b24f107c3b7422d25933897669226107549d20b5cb8d0a3580c5" Nov 03 19:34:16.954780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.945192 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w" Nov 03 19:34:16.968283 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.968248 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b7jj\" (UniqueName: \"kubernetes.io/projected/8933f286-39a0-465f-9b6e-301557aa4157-kube-api-access-5b7jj\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:16.968499 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.968483 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:16.968618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:16.968605 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8933f286-39a0-465f-9b6e-301557aa4157-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:29.049845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:29.049805 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t"] Nov 03 19:34:29.102574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:29.102521 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wmv7t"] Nov 03 19:34:30.753036 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:30.753001 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce49bb21-6e64-4b25-98f2-9bd397d6eb91" path="/var/lib/kubelet/pods/ce49bb21-6e64-4b25-98f2-9bd397d6eb91/volumes" Nov 03 19:34:31.191010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.190962 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq"] Nov 03 19:34:31.241800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.241766 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5zbrq"] Nov 03 19:34:31.303553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.303514 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r"] Nov 03 19:34:31.304029 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.304011 4948 cpu_manager.go:414] "containerMap: removing container" podUID="8933f286-39a0-465f-9b6e-301557aa4157" containerName="revoke-keys" Nov 03 19:34:31.304140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.304031 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8933f286-39a0-465f-9b6e-301557aa4157" containerName="revoke-keys" Nov 03 19:34:31.304208 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.304177 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="8933f286-39a0-465f-9b6e-301557aa4157" containerName="revoke-keys" Nov 03 19:34:31.307607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.307585 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.361733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.361400 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:34:31.361733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.361421 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:34:31.361733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.361414 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:34:31.361733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.361479 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:34:31.362108 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.361782 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r"] Nov 03 19:34:31.425486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.425450 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64nd\" (UniqueName: \"kubernetes.io/projected/0d6b4dab-b56f-429c-a421-491c17f3c43d-kube-api-access-h64nd\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.425670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.425509 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.425771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.425667 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.527247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.527167 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64nd\" (UniqueName: \"kubernetes.io/projected/0d6b4dab-b56f-429c-a421-491c17f3c43d-kube-api-access-h64nd\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.527247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.527212 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.527442 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.527260 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.527922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.527902 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.529655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.529630 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.589097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.589061 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64nd\" (UniqueName: \"kubernetes.io/projected/0d6b4dab-b56f-429c-a421-491c17f3c43d-kube-api-access-h64nd\") pod \"maas-api-revoke-sub-3d4156f1-69s2r\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:31.621809 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:31.621772 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:32.002237 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:32.002205 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r"] Nov 03 19:34:32.002975 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:34:32.002950 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6b4dab_b56f_429c_a421_491c17f3c43d.slice/crio-2b3a3254377841ed02ed203463de46d3a24cc228f5e66c9ba91592dd34e19ce6 WatchSource:0}: Error finding container 2b3a3254377841ed02ed203463de46d3a24cc228f5e66c9ba91592dd34e19ce6: Status 404 returned error can't find the container with id 2b3a3254377841ed02ed203463de46d3a24cc228f5e66c9ba91592dd34e19ce6 Nov 03 19:34:32.038287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:32.038257 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" event={"ID":"0d6b4dab-b56f-429c-a421-491c17f3c43d","Type":"ContainerStarted","Data":"2b3a3254377841ed02ed203463de46d3a24cc228f5e66c9ba91592dd34e19ce6"} Nov 03 19:34:32.753038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:32.753004 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7edfad3-8b08-47d6-92cc-52170b43e4b3" path="/var/lib/kubelet/pods/b7edfad3-8b08-47d6-92cc-52170b43e4b3/volumes" Nov 03 19:34:32.793178 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:34:32.793133 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:34:33.047137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:33.047058 4948 generic.go:356] "Generic (PLEG): container finished" podID="0d6b4dab-b56f-429c-a421-491c17f3c43d" containerID="d79110ad02dcc5c3553cea9f8e901ef6dbf217361b7552975d2112841fb0a10a" exitCode=0 Nov 03 19:34:33.047137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:33.047093 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" event={"ID":"0d6b4dab-b56f-429c-a421-491c17f3c43d","Type":"ContainerDied","Data":"d79110ad02dcc5c3553cea9f8e901ef6dbf217361b7552975d2112841fb0a10a"} Nov 03 19:34:34.194821 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.194795 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:34.255575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.255546 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-openshift-ca\") pod \"0d6b4dab-b56f-429c-a421-491c17f3c43d\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " Nov 03 19:34:34.255754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.255678 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-xks-ca\") pod \"0d6b4dab-b56f-429c-a421-491c17f3c43d\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " Nov 03 19:34:34.255754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.255730 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6b4dab-b56f-429c-a421-491c17f3c43d-kube-api-access-h64nd\" (UniqueName: \"kubernetes.io/projected/0d6b4dab-b56f-429c-a421-491c17f3c43d-kube-api-access-h64nd\") pod \"0d6b4dab-b56f-429c-a421-491c17f3c43d\" (UID: \"0d6b4dab-b56f-429c-a421-491c17f3c43d\") " Nov 03 19:34:34.255986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.255961 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-openshift-ca" pod "0d6b4dab-b56f-429c-a421-491c17f3c43d" (UID: "0d6b4dab-b56f-429c-a421-491c17f3c43d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:34:34.257850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.257820 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-xks-ca" pod "0d6b4dab-b56f-429c-a421-491c17f3c43d" (UID: "0d6b4dab-b56f-429c-a421-491c17f3c43d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:34:34.257956 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.257849 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6b4dab-b56f-429c-a421-491c17f3c43d-kube-api-access-h64nd" pod "0d6b4dab-b56f-429c-a421-491c17f3c43d" (UID: "0d6b4dab-b56f-429c-a421-491c17f3c43d"). InnerVolumeSpecName "kube-api-access-h64nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:34.356906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.356840 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.356906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.356867 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64nd\" (UniqueName: \"kubernetes.io/projected/0d6b4dab-b56f-429c-a421-491c17f3c43d-kube-api-access-h64nd\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:34.356906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:34.356880 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6b4dab-b56f-429c-a421-491c17f3c43d-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:35.069106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:35.069022 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" Nov 03 19:34:35.069106 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:35.069053 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r" event={"ID":"0d6b4dab-b56f-429c-a421-491c17f3c43d","Type":"ContainerDied","Data":"2b3a3254377841ed02ed203463de46d3a24cc228f5e66c9ba91592dd34e19ce6"} Nov 03 19:34:35.069346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:35.069145 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3a3254377841ed02ed203463de46d3a24cc228f5e66c9ba91592dd34e19ce6" Nov 03 19:34:41.050802 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:41.050763 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h"] Nov 03 19:34:41.101591 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:41.101557 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-gg54h"] Nov 03 19:34:42.752773 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:34:42.752743 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8d90cf-8396-4840-a75b-34e54478311a" path="/var/lib/kubelet/pods/3b8d90cf-8396-4840-a75b-34e54478311a/volumes" Nov 03 19:35:01.204479 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:01.204446 4948 scope.go:122] "RemoveContainer" containerID="ef0fb517ace09ea0172c7a155726ddedf958a752b92afdfd1c842ebb67aaed21" Nov 03 19:35:02.793185 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:35:02.793149 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:35:03.592205 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.592167 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd"] Nov 03 19:35:03.592829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.592803 4948 cpu_manager.go:414] "containerMap: removing container" podUID="0d6b4dab-b56f-429c-a421-491c17f3c43d" containerName="revoke-keys" Nov 03 19:35:03.592829 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.592826 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6b4dab-b56f-429c-a421-491c17f3c43d" containerName="revoke-keys" Nov 03 19:35:03.592989 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.592899 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6b4dab-b56f-429c-a421-491c17f3c43d" containerName="revoke-keys" Nov 03 19:35:03.597470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.597449 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663123 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-020814-openshift-default-dockercfg-pcpsz\"" Nov 03 19:35:03.663418 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663390 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-certs\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663507 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663446 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-data\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663570 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istiod-ca-cert\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663626 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-credential-socket\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663720 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663658 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-token\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663780 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-podinfo\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663809 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-envoy\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.663951 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663912 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-socket\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.664012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.663980 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwsz\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-kube-api-access-cwwsz\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.717008 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.716965 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd"] Nov 03 19:35:03.764757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.764690 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-socket\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.764757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.764753 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwsz\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-kube-api-access-cwwsz\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.764798 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-certs\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.764835 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-data\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.764888 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istiod-ca-cert\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.764914 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-credential-socket\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.764940 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-token\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765042 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.765021 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-podinfo\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.765055 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-envoy\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.765175 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-socket\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.765265 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-certs\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.765380 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-data\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765624 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.765494 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-credential-socket\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.765792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.765772 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istiod-ca-cert\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.767253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.767231 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-envoy\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.767629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.767610 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-podinfo\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.827268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.827234 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-token\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.827628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.827312 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwsz\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-kube-api-access-cwwsz\") pod \"e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:03.910664 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:03.910569 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:04.342538 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:35:04.342510 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c5d9bf_f941_421e_ab76_cdd44dfca7a6.slice/crio-5b6ad2486dfc2b939b864eea4dd2dd47f50734897f7b3b6f076421df63ca24c3 WatchSource:0}: Error finding container 5b6ad2486dfc2b939b864eea4dd2dd47f50734897f7b3b6f076421df63ca24c3: Status 404 returned error can't find the container with id 5b6ad2486dfc2b939b864eea4dd2dd47f50734897f7b3b6f076421df63ca24c3 Nov 03 19:35:04.342789 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:04.342763 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd"] Nov 03 19:35:04.345000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:04.344960 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:04.345105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:04.345037 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:04.345105 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:04.345073 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:05.336887 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:05.336848 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" event={"ID":"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6","Type":"ContainerStarted","Data":"043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd"} Nov 03 19:35:05.337355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:05.336902 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" event={"ID":"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6","Type":"ContainerStarted","Data":"5b6ad2486dfc2b939b864eea4dd2dd47f50734897f7b3b6f076421df63ca24c3"} Nov 03 19:35:05.515677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:05.515619 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" podStartSLOduration=2.5156052129999997 podStartE2EDuration="2.515605213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:05.51325487 +0000 UTC m=+3434.996250784" watchObservedRunningTime="2026-11-03 19:35:05.515605213 +0000 UTC m=+3434.998601156" Nov 03 19:35:05.910776 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:05.910736 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:05.915884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:05.915857 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:06.344018 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:06.343977 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:06.345222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:06.345201 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:12.010853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.010812 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf"] Nov 03 19:35:12.015141 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.015117 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.071486 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.071452 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf"] Nov 03 19:35:12.126832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.123998 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-020814\"" Nov 03 19:35:12.126832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.124084 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ee66d572-dockercfg-h6hbx\"" Nov 03 19:35:12.154690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.154657 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9z2\" (UniqueName: \"kubernetes.io/projected/b6f41080-7e48-4aa7-849f-e64d113f21c9-kube-api-access-9b9z2\") pod \"payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.154891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.154808 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f41080-7e48-4aa7-849f-e64d113f21c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.178115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.178085 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4"] Nov 03 19:35:12.181788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.181771 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.214957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.214925 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4"] Nov 03 19:35:12.255682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.255645 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a84764-3ac8-4547-b0cc-109fbf93ba79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.255896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.255783 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f41080-7e48-4aa7-849f-e64d113f21c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.255896 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.255844 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4nf\" (UniqueName: \"kubernetes.io/projected/62a84764-3ac8-4547-b0cc-109fbf93ba79-kube-api-access-4g4nf\") pod \"payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.255981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.255909 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9z2\" (UniqueName: \"kubernetes.io/projected/b6f41080-7e48-4aa7-849f-e64d113f21c9-kube-api-access-9b9z2\") pod \"payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.256653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.256631 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f41080-7e48-4aa7-849f-e64d113f21c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.318339 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.318260 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9z2\" (UniqueName: \"kubernetes.io/projected/b6f41080-7e48-4aa7-849f-e64d113f21c9-kube-api-access-9b9z2\") pod \"payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.329056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.329032 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:12.357485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.357452 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a84764-3ac8-4547-b0cc-109fbf93ba79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.357639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.357579 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4nf\" (UniqueName: \"kubernetes.io/projected/62a84764-3ac8-4547-b0cc-109fbf93ba79-kube-api-access-4g4nf\") pod \"payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.358182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.358153 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a84764-3ac8-4547-b0cc-109fbf93ba79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.420641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.420605 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4nf\" (UniqueName: \"kubernetes.io/projected/62a84764-3ac8-4547-b0cc-109fbf93ba79-kube-api-access-4g4nf\") pod \"payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.494496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.494463 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:12.709674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.709645 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf"] Nov 03 19:35:12.710483 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:35:12.710451 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f41080_7e48_4aa7_849f_e64d113f21c9.slice/crio-6016134da30176cfe68f9e9e90c7c471c4fd701731667dc8332cbaa6d54cdfdb WatchSource:0}: Error finding container 6016134da30176cfe68f9e9e90c7c471c4fd701731667dc8332cbaa6d54cdfdb: Status 404 returned error can't find the container with id 6016134da30176cfe68f9e9e90c7c471c4fd701731667dc8332cbaa6d54cdfdb Nov 03 19:35:12.874481 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:12.874450 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4"] Nov 03 19:35:12.875008 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:35:12.874983 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a84764_3ac8_4547_b0cc_109fbf93ba79.slice/crio-cd24fa7f44788bdd5d1070f11a3372c6e56036a1409f9af8cef61789a6c92000 WatchSource:0}: Error finding container cd24fa7f44788bdd5d1070f11a3372c6e56036a1409f9af8cef61789a6c92000: Status 404 returned error can't find the container with id cd24fa7f44788bdd5d1070f11a3372c6e56036a1409f9af8cef61789a6c92000 Nov 03 19:35:13.389488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.389445 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" event={"ID":"b6f41080-7e48-4aa7-849f-e64d113f21c9","Type":"ContainerStarted","Data":"733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29"} Nov 03 19:35:13.389991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.389514 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:13.389991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.389538 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" event={"ID":"b6f41080-7e48-4aa7-849f-e64d113f21c9","Type":"ContainerStarted","Data":"6016134da30176cfe68f9e9e90c7c471c4fd701731667dc8332cbaa6d54cdfdb"} Nov 03 19:35:13.391191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.391162 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" event={"ID":"62a84764-3ac8-4547-b0cc-109fbf93ba79","Type":"ContainerStarted","Data":"14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0"} Nov 03 19:35:13.391329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.391220 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:13.391329 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.391243 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" event={"ID":"62a84764-3ac8-4547-b0cc-109fbf93ba79","Type":"ContainerStarted","Data":"cd24fa7f44788bdd5d1070f11a3372c6e56036a1409f9af8cef61789a6c92000"} Nov 03 19:35:13.507667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.507612 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" podStartSLOduration=2.507593685 podStartE2EDuration="2.507593685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:13.506943083 +0000 UTC m=+3442.989938997" watchObservedRunningTime="2026-11-03 19:35:13.507593685 +0000 UTC m=+3442.990589599" Nov 03 19:35:13.629048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:13.628983 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" podStartSLOduration=1.628965396 podStartE2EDuration="1.628965396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:13.625898356 +0000 UTC m=+3443.108894270" watchObservedRunningTime="2026-11-03 19:35:13.628965396 +0000 UTC m=+3443.111961309" Nov 03 19:35:14.308711 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.308666 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w"] Nov 03 19:35:14.312371 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.312354 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.369059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.369027 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w"] Nov 03 19:35:14.376509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.376478 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-metrics-certs\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.376509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.376511 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d454811-b798-4ed5-ac88-111921f8aa58-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.376755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.376667 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthg2\" (UniqueName: \"kubernetes.io/projected/7d454811-b798-4ed5-ac88-111921f8aa58-kube-api-access-gthg2\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.376755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.376726 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.416061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.416027 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:35:14.416061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.416057 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\"" Nov 03 19:35:14.416615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.416052 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-020814-serving-cert\"" Nov 03 19:35:14.416615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.416102 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:35:14.416615 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.416159 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 19:35:14.477728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.477661 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-metrics-certs\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.477923 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.477737 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d454811-b798-4ed5-ac88-111921f8aa58-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.477993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.477923 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthg2\" (UniqueName: \"kubernetes.io/projected/7d454811-b798-4ed5-ac88-111921f8aa58-kube-api-access-gthg2\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.477993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.477980 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.478133 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:35:14.478104 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-serving-cert: secret "maas-api-e2e-worker-single-main-020814-serving-cert" not found Nov 03 19:35:14.478225 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:35:14.478210 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls podName:7d454811-b798-4ed5-ac88-111921f8aa58 nodeName:}" failed. No retries permitted until 2026-11-03 19:35:14.978183066 +0000 UTC m=+3444.461178978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls") pod "maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" (UID: "7d454811-b798-4ed5-ac88-111921f8aa58") : secret "maas-api-e2e-worker-single-main-020814-serving-cert" not found Nov 03 19:35:14.478523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.478505 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d454811-b798-4ed5-ac88-111921f8aa58-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.480407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.480383 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-metrics-certs\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.543679 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.543637 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthg2\" (UniqueName: \"kubernetes.io/projected/7d454811-b798-4ed5-ac88-111921f8aa58-kube-api-access-gthg2\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.984770 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.984729 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:14.987034 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:14.987006 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:15.226692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:15.226652 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:15.605408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:15.605376 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w"] Nov 03 19:35:15.605789 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:35:15.605669 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d454811_b798_4ed5_ac88_111921f8aa58.slice/crio-348da4cd46d15fbdab038dcbb6b42a4c69cd8972b1d8d0802c4764641d334d38 WatchSource:0}: Error finding container 348da4cd46d15fbdab038dcbb6b42a4c69cd8972b1d8d0802c4764641d334d38: Status 404 returned error can't find the container with id 348da4cd46d15fbdab038dcbb6b42a4c69cd8972b1d8d0802c4764641d334d38 Nov 03 19:35:15.607263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:15.607244 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:35:16.421445 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:16.421411 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" event={"ID":"7d454811-b798-4ed5-ac88-111921f8aa58","Type":"ContainerStarted","Data":"348da4cd46d15fbdab038dcbb6b42a4c69cd8972b1d8d0802c4764641d334d38"} Nov 03 19:35:17.430006 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:17.429957 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" event={"ID":"7d454811-b798-4ed5-ac88-111921f8aa58","Type":"ContainerStarted","Data":"e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87"} Nov 03 19:35:17.430496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:17.430046 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:17.549574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:17.549516 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" podStartSLOduration=2.705953675 podStartE2EDuration="3.549501138s" totalImagesPullingTime="843.547463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:17.54908045 +0000 UTC m=+3447.032076364" watchObservedRunningTime="2026-11-03 19:35:17.549501138 +0000 UTC m=+3447.032497091" Nov 03 19:35:23.441768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:23.441739 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:35:24.401596 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:24.401562 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:35:24.401832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:24.401633 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:35:30.050670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:30.050622 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb"] Nov 03 19:35:30.101483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:30.101447 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bfzzb"] Nov 03 19:35:30.752762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:30.752657 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75ac79a-b90c-445e-b357-107063b75271" path="/var/lib/kubelet/pods/c75ac79a-b90c-445e-b357-107063b75271/volumes" Nov 03 19:35:32.560049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:32.560014 4948 generic.go:356] "Generic (PLEG): container finished" podID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerID="043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd" exitCode=0 Nov 03 19:35:32.560608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:32.560088 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" event={"ID":"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6","Type":"ContainerDied","Data":"043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd"} Nov 03 19:35:32.560728 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:32.560615 4948 scope.go:122] "RemoveContainer" containerID="043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd" Nov 03 19:35:32.561874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:32.561844 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:32.562002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:32.561927 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:32.562002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:32.561975 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:32.805062 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:35:32.803496 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:35:33.587144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:33.587116 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" event={"ID":"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6","Type":"ContainerStarted","Data":"364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c"} Nov 03 19:35:33.911799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:33.911682 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:33.913192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:33.913161 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Nov 03 19:35:33.913345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:33.913216 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Nov 03 19:35:34.911655 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:34.911618 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Nov 03 19:35:34.912130 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:34.911677 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Nov 03 19:35:35.911996 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:35.911960 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Nov 03 19:35:35.912408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:35.912028 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Nov 03 19:35:36.344778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:36.344693 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:36.911382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:36.911342 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Nov 03 19:35:36.911660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:36.911406 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Nov 03 19:35:37.911886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:37.911851 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Nov 03 19:35:37.912338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:37.911913 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Nov 03 19:35:38.269177 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.269126 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8"] Nov 03 19:35:38.280282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.280253 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.322969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.322937 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8"] Nov 03 19:35:38.330669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.330637 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-tmp-dir\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.330853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.330680 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-dshm\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.330853 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.330830 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-model-cache\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.331003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.330892 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9547aa6-e95d-4967-8b78-029d25c09d4a-tls-certs\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.331003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.330941 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-home\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.331003 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.330990 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbb8\" (UniqueName: \"kubernetes.io/projected/f9547aa6-e95d-4967-8b78-029d25c09d4a-kube-api-access-qqbb8\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.376097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.376059 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-020814\"/\"openshift-service-ca.crt\"" Nov 03 19:35:38.384639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.384613 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-020814\"/\"simulator-single-main-020814-kserve-self-signed-certs\"" Nov 03 19:35:38.384751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.384637 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-020814\"/\"kube-root-ca.crt\"" Nov 03 19:35:38.384751 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.384676 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-020814\"/\"default-dockercfg-prt7s\"" Nov 03 19:35:38.431924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.431878 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-tmp-dir\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.431924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.431916 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-dshm\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.432206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.431960 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-model-cache\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.432206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.431980 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9547aa6-e95d-4967-8b78-029d25c09d4a-tls-certs\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.432206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.432006 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-home\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.432206 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.432038 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbb8\" (UniqueName: \"kubernetes.io/projected/f9547aa6-e95d-4967-8b78-029d25c09d4a-kube-api-access-qqbb8\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.432386 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.432335 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-tmp-dir\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.432386 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.432358 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-model-cache\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.432465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.432410 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-home\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.434325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.434296 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-dshm\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.434604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.434587 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9547aa6-e95d-4967-8b78-029d25c09d4a-tls-certs\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.494187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.494151 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbb8\" (UniqueName: \"kubernetes.io/projected/f9547aa6-e95d-4967-8b78-029d25c09d4a-kube-api-access-qqbb8\") pod \"simulator-single-main-020814-kserve-7b6998fb85-fmtz8\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.607457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.607359 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:38.915677 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.915593 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:38.916580 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:38.916560 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:35:39.004360 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:35:39.004323 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9547aa6_e95d_4967_8b78_029d25c09d4a.slice/crio-6fd37435f812ca842d5071a9dc8d4f0591fdd85a84d6753215522873d2af49f8 WatchSource:0}: Error finding container 6fd37435f812ca842d5071a9dc8d4f0591fdd85a84d6753215522873d2af49f8: Status 404 returned error can't find the container with id 6fd37435f812ca842d5071a9dc8d4f0591fdd85a84d6753215522873d2af49f8 Nov 03 19:35:39.023204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:39.023177 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8"] Nov 03 19:35:39.632519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:39.632444 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" event={"ID":"f9547aa6-e95d-4967-8b78-029d25c09d4a","Type":"ContainerStarted","Data":"a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e"} Nov 03 19:35:39.632519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:39.632524 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" event={"ID":"f9547aa6-e95d-4967-8b78-029d25c09d4a","Type":"ContainerStarted","Data":"6fd37435f812ca842d5071a9dc8d4f0591fdd85a84d6753215522873d2af49f8"} Nov 03 19:35:39.752196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:39.752141 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" podStartSLOduration=1.752128027 podStartE2EDuration="1.752128027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:39.751311592 +0000 UTC m=+3469.234307506" watchObservedRunningTime="2026-11-03 19:35:39.752128027 +0000 UTC m=+3469.235123994" Nov 03 19:35:40.607640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:40.607605 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:48.608245 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:48.608200 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:48.621740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:48.621713 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:35:48.706468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:35:48.706439 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:36:01.229464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:01.229433 4948 scope.go:122] "RemoveContainer" containerID="3a9545f8e193baecbbd6c685d9afc0f13f1126109d86566289ad0ed5b1dcf9bb" Nov 03 19:36:01.238986 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:01.238962 4948 scope.go:122] "RemoveContainer" containerID="22ff37a4bbece740425e184098c429515ff5ed91b939081225efc18ac6e925aa" Nov 03 19:36:02.795296 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:36:02.795260 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:36:05.049882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:05.049844 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6"] Nov 03 19:36:05.100340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:05.100309 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8rqq6"] Nov 03 19:36:06.753547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:06.753504 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42cbe7f-5825-435d-89e9-7398b7ee9e40" path="/var/lib/kubelet/pods/c42cbe7f-5825-435d-89e9-7398b7ee9e40/volumes" Nov 03 19:36:32.794391 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:36:32.794352 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:36:35.052905 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:35.052864 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv"] Nov 03 19:36:35.103695 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:35.103650 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mtxzv"] Nov 03 19:36:36.752902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:36.752864 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac82fbbf-ef97-425e-a948-c20cc46a4dae" path="/var/lib/kubelet/pods/ac82fbbf-ef97-425e-a948-c20cc46a4dae/volumes" Nov 03 19:36:40.169651 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:40.169606 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8"] Nov 03 19:36:46.223265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:46.223220 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w"] Nov 03 19:36:46.223857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:46.223478 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" podUID="7d454811-b798-4ed5-ac88-111921f8aa58" containerName="maas-api" containerID="cri-o://e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87" gracePeriod=30 Nov 03 19:36:46.653787 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:46.653682 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4"] Nov 03 19:36:46.704951 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:46.704914 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf"] Nov 03 19:36:47.487362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.487334 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:36:47.552644 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.552611 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-metrics-certs\") pod \"7d454811-b798-4ed5-ac88-111921f8aa58\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " Nov 03 19:36:47.552873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.552661 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d454811-b798-4ed5-ac88-111921f8aa58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d454811-b798-4ed5-ac88-111921f8aa58-openshift-service-ca\") pod \"7d454811-b798-4ed5-ac88-111921f8aa58\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " Nov 03 19:36:47.552873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.552773 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d454811-b798-4ed5-ac88-111921f8aa58-kube-api-access-gthg2\" (UniqueName: \"kubernetes.io/projected/7d454811-b798-4ed5-ac88-111921f8aa58-kube-api-access-gthg2\") pod \"7d454811-b798-4ed5-ac88-111921f8aa58\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " Nov 03 19:36:47.552873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.552804 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls\") pod \"7d454811-b798-4ed5-ac88-111921f8aa58\" (UID: \"7d454811-b798-4ed5-ac88-111921f8aa58\") " Nov 03 19:36:47.553170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.553143 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d454811-b798-4ed5-ac88-111921f8aa58-openshift-service-ca" pod "7d454811-b798-4ed5-ac88-111921f8aa58" (UID: "7d454811-b798-4ed5-ac88-111921f8aa58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:36:47.555076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.555036 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-metrics-certs" pod "7d454811-b798-4ed5-ac88-111921f8aa58" (UID: "7d454811-b798-4ed5-ac88-111921f8aa58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:47.555076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.555068 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d454811-b798-4ed5-ac88-111921f8aa58-kube-api-access-gthg2" pod "7d454811-b798-4ed5-ac88-111921f8aa58" (UID: "7d454811-b798-4ed5-ac88-111921f8aa58"). InnerVolumeSpecName "kube-api-access-gthg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:47.555262 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.555118 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls" pod "7d454811-b798-4ed5-ac88-111921f8aa58" (UID: "7d454811-b798-4ed5-ac88-111921f8aa58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:47.654043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.653950 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:47.654043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.653990 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d454811-b798-4ed5-ac88-111921f8aa58-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:47.654043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.654005 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthg2\" (UniqueName: \"kubernetes.io/projected/7d454811-b798-4ed5-ac88-111921f8aa58-kube-api-access-gthg2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:47.654043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:47.654021 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d454811-b798-4ed5-ac88-111921f8aa58-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:48.090450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.090414 4948 generic.go:356] "Generic (PLEG): container finished" podID="7d454811-b798-4ed5-ac88-111921f8aa58" containerID="e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87" exitCode=0 Nov 03 19:36:48.090642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.090478 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" Nov 03 19:36:48.090642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.090505 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" event={"ID":"7d454811-b798-4ed5-ac88-111921f8aa58","Type":"ContainerDied","Data":"e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87"} Nov 03 19:36:48.090642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.090589 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w" event={"ID":"7d454811-b798-4ed5-ac88-111921f8aa58","Type":"ContainerDied","Data":"348da4cd46d15fbdab038dcbb6b42a4c69cd8972b1d8d0802c4764641d334d38"} Nov 03 19:36:48.090642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.090616 4948 scope.go:122] "RemoveContainer" containerID="e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87" Nov 03 19:36:48.101357 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.101335 4948 scope.go:122] "RemoveContainer" containerID="e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87" Nov 03 19:36:48.101643 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:36:48.101623 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87\": container with ID starting with e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87 not found: ID does not exist" containerID="e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87" Nov 03 19:36:48.101691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.101652 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87"} err="failed to get container status \"e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87\": rpc error: code = NotFound desc = could not find container \"e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87\": container with ID starting with e50476979d2ae261a9a27a8de3383298e9c77bb26588ff6b8d49a1fcd2c64d87 not found: ID does not exist" Nov 03 19:36:48.258271 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.258234 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w"] Nov 03 19:36:48.276772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.276745 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-020814-747df7fb9-7lf5w"] Nov 03 19:36:48.752814 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:48.752780 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d454811-b798-4ed5-ac88-111921f8aa58" path="/var/lib/kubelet/pods/7d454811-b798-4ed5-ac88-111921f8aa58/volumes" Nov 03 19:36:51.654054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:51.654000 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" podUID="62a84764-3ac8-4547-b0cc-109fbf93ba79" containerName="payload-processing" containerID="cri-o://14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0" gracePeriod=25 Nov 03 19:36:51.705435 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:51.705393 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" podUID="b6f41080-7e48-4aa7-849f-e64d113f21c9" containerName="payload-pre-processing" containerID="cri-o://733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29" gracePeriod=25 Nov 03 19:36:54.400069 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:54.400023 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" podUID="b6f41080-7e48-4aa7-849f-e64d113f21c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:36:54.400468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:54.400023 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" podUID="62a84764-3ac8-4547-b0cc-109fbf93ba79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:36:55.190785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.190746 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" podUID="f9547aa6-e95d-4967-8b78-029d25c09d4a" containerName="main" containerID="cri-o://a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e" gracePeriod=45 Nov 03 19:36:55.459517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.459487 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:36:55.530752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.530717 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-model-cache\") pod \"f9547aa6-e95d-4967-8b78-029d25c09d4a\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " Nov 03 19:36:55.530957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.530780 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9547aa6-e95d-4967-8b78-029d25c09d4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9547aa6-e95d-4967-8b78-029d25c09d4a-tls-certs\") pod \"f9547aa6-e95d-4967-8b78-029d25c09d4a\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " Nov 03 19:36:55.530957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.530812 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-home\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-home\") pod \"f9547aa6-e95d-4967-8b78-029d25c09d4a\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " Nov 03 19:36:55.530957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.530861 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9547aa6-e95d-4967-8b78-029d25c09d4a-kube-api-access-qqbb8\" (UniqueName: \"kubernetes.io/projected/f9547aa6-e95d-4967-8b78-029d25c09d4a-kube-api-access-qqbb8\") pod \"f9547aa6-e95d-4967-8b78-029d25c09d4a\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " Nov 03 19:36:55.531137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.530975 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-tmp-dir\") pod \"f9547aa6-e95d-4967-8b78-029d25c09d4a\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " Nov 03 19:36:55.531137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.531006 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-dshm\") pod \"f9547aa6-e95d-4967-8b78-029d25c09d4a\" (UID: \"f9547aa6-e95d-4967-8b78-029d25c09d4a\") " Nov 03 19:36:55.531137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.531042 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-model-cache" pod "f9547aa6-e95d-4967-8b78-029d25c09d4a" (UID: "f9547aa6-e95d-4967-8b78-029d25c09d4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:55.531281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.531139 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-home" pod "f9547aa6-e95d-4967-8b78-029d25c09d4a" (UID: "f9547aa6-e95d-4967-8b78-029d25c09d4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:55.531281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.531262 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-tmp-dir" pod "f9547aa6-e95d-4967-8b78-029d25c09d4a" (UID: "f9547aa6-e95d-4967-8b78-029d25c09d4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:55.531456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.531430 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:55.531456 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.531454 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:55.531660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.531467 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:55.533147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.533124 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-dshm" pod "f9547aa6-e95d-4967-8b78-029d25c09d4a" (UID: "f9547aa6-e95d-4967-8b78-029d25c09d4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:36:55.533252 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.533183 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9547aa6-e95d-4967-8b78-029d25c09d4a-tls-certs" pod "f9547aa6-e95d-4967-8b78-029d25c09d4a" (UID: "f9547aa6-e95d-4967-8b78-029d25c09d4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:55.533302 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.533253 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9547aa6-e95d-4967-8b78-029d25c09d4a-kube-api-access-qqbb8" pod "f9547aa6-e95d-4967-8b78-029d25c09d4a" (UID: "f9547aa6-e95d-4967-8b78-029d25c09d4a"). InnerVolumeSpecName "kube-api-access-qqbb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:55.633139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.633088 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9547aa6-e95d-4967-8b78-029d25c09d4a-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:55.633316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.633142 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbb8\" (UniqueName: \"kubernetes.io/projected/f9547aa6-e95d-4967-8b78-029d25c09d4a-kube-api-access-qqbb8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:55.633316 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:55.633162 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9547aa6-e95d-4967-8b78-029d25c09d4a-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:56.147336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.147297 4948 generic.go:356] "Generic (PLEG): container finished" podID="f9547aa6-e95d-4967-8b78-029d25c09d4a" containerID="a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e" exitCode=0 Nov 03 19:36:56.147529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.147349 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" Nov 03 19:36:56.147529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.147374 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" event={"ID":"f9547aa6-e95d-4967-8b78-029d25c09d4a","Type":"ContainerDied","Data":"a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e"} Nov 03 19:36:56.147529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.147418 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8" event={"ID":"f9547aa6-e95d-4967-8b78-029d25c09d4a","Type":"ContainerDied","Data":"6fd37435f812ca842d5071a9dc8d4f0591fdd85a84d6753215522873d2af49f8"} Nov 03 19:36:56.147529 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.147437 4948 scope.go:122] "RemoveContainer" containerID="a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e" Nov 03 19:36:56.158398 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.158378 4948 scope.go:122] "RemoveContainer" containerID="a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e" Nov 03 19:36:56.158710 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:36:56.158682 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e\": container with ID starting with a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e not found: ID does not exist" containerID="a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e" Nov 03 19:36:56.158784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.158724 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e"} err="failed to get container status \"a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e\": rpc error: code = NotFound desc = could not find container \"a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e\": container with ID starting with a902fe92dcbf6fd546ad1563b71f68506d96f9b37f2421a1963c0cc55f7fb94e not found: ID does not exist" Nov 03 19:36:56.315167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.315129 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8"] Nov 03 19:36:56.366660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.366625 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-020814/simulator-single-main-020814-kserve-7b6998fb85-fmtz8"] Nov 03 19:36:56.752994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:56.752955 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9547aa6-e95d-4967-8b78-029d25c09d4a" path="/var/lib/kubelet/pods/f9547aa6-e95d-4967-8b78-029d25c09d4a/volumes" Nov 03 19:36:58.644738 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:58.644694 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd"] Nov 03 19:36:58.645209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:36:58.644905 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" containerID="cri-o://364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c" gracePeriod=30 Nov 03 19:37:00.233527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.233489 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r"] Nov 03 19:37:00.284495 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.284458 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69s2r"] Nov 03 19:37:00.350546 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.350509 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w"] Nov 03 19:37:00.351004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.350990 4948 cpu_manager.go:414] "containerMap: removing container" podUID="7d454811-b798-4ed5-ac88-111921f8aa58" containerName="maas-api" Nov 03 19:37:00.351056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.351005 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d454811-b798-4ed5-ac88-111921f8aa58" containerName="maas-api" Nov 03 19:37:00.351056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.351034 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f9547aa6-e95d-4967-8b78-029d25c09d4a" containerName="main" Nov 03 19:37:00.351056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.351039 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9547aa6-e95d-4967-8b78-029d25c09d4a" containerName="main" Nov 03 19:37:00.351163 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.351107 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9547aa6-e95d-4967-8b78-029d25c09d4a" containerName="main" Nov 03 19:37:00.351163 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.351120 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d454811-b798-4ed5-ac88-111921f8aa58" containerName="maas-api" Nov 03 19:37:00.355583 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.355565 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.408946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.408909 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:37:00.409145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.408992 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:37:00.409145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.409099 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:37:00.409145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.409108 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:37:00.409729 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.409676 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w"] Nov 03 19:37:00.487968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.487884 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.487968 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.487951 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqtm8\" (UniqueName: \"kubernetes.io/projected/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-kube-api-access-wqtm8\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.488189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.488123 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.589143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.589105 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.589336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.589260 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.589410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.589364 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtm8\" (UniqueName: \"kubernetes.io/projected/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-kube-api-access-wqtm8\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.589958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.589930 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.591489 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.591465 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.651170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.651142 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtm8\" (UniqueName: \"kubernetes.io/projected/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-kube-api-access-wqtm8\") pod \"maas-api-revoke-sub-3d4156f1-m964w\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.669618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.669588 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:00.765488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:00.765414 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6b4dab-b56f-429c-a421-491c17f3c43d" path="/var/lib/kubelet/pods/0d6b4dab-b56f-429c-a421-491c17f3c43d/volumes" Nov 03 19:37:01.049321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:01.049280 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w"] Nov 03 19:37:01.050252 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:37:01.050226 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bcbc46_49a4_4492_b3f5_a1246e9ad4ea.slice/crio-3461c15754c15c769fd8844b3693d7402c26489fd71b505e3aa438c3f6b95a94 WatchSource:0}: Error finding container 3461c15754c15c769fd8844b3693d7402c26489fd71b505e3aa438c3f6b95a94: Status 404 returned error can't find the container with id 3461c15754c15c769fd8844b3693d7402c26489fd71b505e3aa438c3f6b95a94 Nov 03 19:37:01.182014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:01.181977 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" event={"ID":"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea","Type":"ContainerStarted","Data":"3461c15754c15c769fd8844b3693d7402c26489fd71b505e3aa438c3f6b95a94"} Nov 03 19:37:01.268346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:01.268313 4948 scope.go:122] "RemoveContainer" containerID="a51b2525aea82e2e82a38b7c1d209529c41f36d8340eaa2af4712c2eb8ae3d15" Nov 03 19:37:02.189901 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:02.189865 4948 generic.go:356] "Generic (PLEG): container finished" podID="10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" containerID="4cc0be300ec33b683505f31c5afa73abc04b2e67686aafa491f078b9c2e0a2a3" exitCode=0 Nov 03 19:37:02.190095 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:02.189932 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" event={"ID":"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea","Type":"ContainerDied","Data":"4cc0be300ec33b683505f31c5afa73abc04b2e67686aafa491f078b9c2e0a2a3"} Nov 03 19:37:02.793522 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:37:02.793481 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:37:03.343993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.343967 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:03.419194 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.419162 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-openshift-ca\") pod \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " Nov 03 19:37:03.419379 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.419298 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-xks-ca\") pod \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " Nov 03 19:37:03.419379 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.419357 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-kube-api-access-wqtm8\" (UniqueName: \"kubernetes.io/projected/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-kube-api-access-wqtm8\") pod \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\" (UID: \"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea\") " Nov 03 19:37:03.419557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.419532 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-openshift-ca" pod "10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" (UID: "10bcbc46-49a4-4492-b3f5-a1246e9ad4ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:03.421431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.421412 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-xks-ca" pod "10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" (UID: "10bcbc46-49a4-4492-b3f5-a1246e9ad4ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:03.421578 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.421558 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-kube-api-access-wqtm8" pod "10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" (UID: "10bcbc46-49a4-4492-b3f5-a1246e9ad4ea"). InnerVolumeSpecName "kube-api-access-wqtm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:03.521012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.520915 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqtm8\" (UniqueName: \"kubernetes.io/projected/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-kube-api-access-wqtm8\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:03.521012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.520951 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:03.521012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.520966 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:03.903269 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:03.903244 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:37:04.026431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026399 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-certs\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026464 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-credential-socket\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026495 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-data\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026535 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-podinfo\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026591 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istiod-ca-cert\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026649 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026638 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-socket\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026685 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-kube-api-access-cwwsz\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-kube-api-access-cwwsz\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026812 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-envoy\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.026962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.026855 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-token\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-token\") pod \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\" (UID: \"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6\") " Nov 03 19:37:04.027162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.027133 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-socket" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:04.027961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.027384 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-certs" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:04.028135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.028109 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istiod-ca-cert" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:04.028212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.028174 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-data" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:04.028600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.028572 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-credential-socket" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:04.031136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.031093 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-envoy" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:04.031253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.031166 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-token" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:04.031327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.031280 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-podinfo" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:37:04.032744 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.032717 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-kube-api-access-cwwsz" pod "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" (UID: "c0c5d9bf-f941-421e-ab76-cdd44dfca7a6"). InnerVolumeSpecName "kube-api-access-cwwsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128147 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128175 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128184 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128195 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128203 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128212 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwwsz\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-kube-api-access-cwwsz\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128221 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128230 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.128232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.128240 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:04.203353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.203322 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" Nov 03 19:37:04.203519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.203359 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w" event={"ID":"10bcbc46-49a4-4492-b3f5-a1246e9ad4ea","Type":"ContainerDied","Data":"3461c15754c15c769fd8844b3693d7402c26489fd71b505e3aa438c3f6b95a94"} Nov 03 19:37:04.203519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.203409 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3461c15754c15c769fd8844b3693d7402c26489fd71b505e3aa438c3f6b95a94" Nov 03 19:37:04.205310 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.205281 4948 generic.go:356] "Generic (PLEG): container finished" podID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerID="364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c" exitCode=0 Nov 03 19:37:04.205430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.205332 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" Nov 03 19:37:04.205430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.205338 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" event={"ID":"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6","Type":"ContainerDied","Data":"364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c"} Nov 03 19:37:04.205430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.205382 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd" event={"ID":"c0c5d9bf-f941-421e-ab76-cdd44dfca7a6","Type":"ContainerDied","Data":"5b6ad2486dfc2b939b864eea4dd2dd47f50734897f7b3b6f076421df63ca24c3"} Nov 03 19:37:04.205430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.205412 4948 scope.go:122] "RemoveContainer" containerID="364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c" Nov 03 19:37:04.216947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.216928 4948 scope.go:122] "RemoveContainer" containerID="043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd" Nov 03 19:37:04.225752 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.225720 4948 scope.go:122] "RemoveContainer" containerID="364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c" Nov 03 19:37:04.226024 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:37:04.226005 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c\": container with ID starting with 364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c not found: ID does not exist" containerID="364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c" Nov 03 19:37:04.226075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.226032 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c"} err="failed to get container status \"364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c\": rpc error: code = NotFound desc = could not find container \"364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c\": container with ID starting with 364aab9d68cdf842a5f7b4a43c7342c67c73eca7f14afd2c36d1a7f6994a090c not found: ID does not exist" Nov 03 19:37:04.226075 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.226049 4948 scope.go:122] "RemoveContainer" containerID="043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd" Nov 03 19:37:04.226286 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:37:04.226267 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd\": container with ID starting with 043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd not found: ID does not exist" containerID="043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd" Nov 03 19:37:04.226325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.226294 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd"} err="failed to get container status \"043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd\": rpc error: code = NotFound desc = could not find container \"043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd\": container with ID starting with 043803e47b9e85946d6588a5075d6c6fec7625a58d159e1137f6109d8c56dfdd not found: ID does not exist" Nov 03 19:37:04.375122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.375083 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd"] Nov 03 19:37:04.379014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.378941 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-020814-openshift-default-75d7fd8777hwxkd"] Nov 03 19:37:04.399450 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.399418 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" podUID="b6f41080-7e48-4aa7-849f-e64d113f21c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:37:04.399582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.399418 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" podUID="62a84764-3ac8-4547-b0cc-109fbf93ba79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:37:04.752354 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:04.752321 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" path="/var/lib/kubelet/pods/c0c5d9bf-f941-421e-ab76-cdd44dfca7a6/volumes" Nov 03 19:37:06.051485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:06.051447 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59"] Nov 03 19:37:06.102132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:06.102095 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zbs59"] Nov 03 19:37:06.752650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:06.752618 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ca72c1-73e1-419f-89f0-06a762361f6f" path="/var/lib/kubelet/pods/16ca72c1-73e1-419f-89f0-06a762361f6f/volumes" Nov 03 19:37:12.005551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.005518 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:37:12.019317 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.019296 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:37:12.100827 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.100794 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f41080-7e48-4aa7-849f-e64d113f21c9-kube-api-access-9b9z2\" (UniqueName: \"kubernetes.io/projected/b6f41080-7e48-4aa7-849f-e64d113f21c9-kube-api-access-9b9z2\") pod \"b6f41080-7e48-4aa7-849f-e64d113f21c9\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " Nov 03 19:37:12.101026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.100873 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a84764-3ac8-4547-b0cc-109fbf93ba79-kube-api-access-4g4nf\" (UniqueName: \"kubernetes.io/projected/62a84764-3ac8-4547-b0cc-109fbf93ba79-kube-api-access-4g4nf\") pod \"62a84764-3ac8-4547-b0cc-109fbf93ba79\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " Nov 03 19:37:12.101026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.100939 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f41080-7e48-4aa7-849f-e64d113f21c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f41080-7e48-4aa7-849f-e64d113f21c9-plugins-config-volume\") pod \"b6f41080-7e48-4aa7-849f-e64d113f21c9\" (UID: \"b6f41080-7e48-4aa7-849f-e64d113f21c9\") " Nov 03 19:37:12.101026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.100958 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a84764-3ac8-4547-b0cc-109fbf93ba79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a84764-3ac8-4547-b0cc-109fbf93ba79-plugins-config-volume\") pod \"62a84764-3ac8-4547-b0cc-109fbf93ba79\" (UID: \"62a84764-3ac8-4547-b0cc-109fbf93ba79\") " Nov 03 19:37:12.101352 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.101327 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f41080-7e48-4aa7-849f-e64d113f21c9-plugins-config-volume" pod "b6f41080-7e48-4aa7-849f-e64d113f21c9" (UID: "b6f41080-7e48-4aa7-849f-e64d113f21c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:12.101457 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.101430 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a84764-3ac8-4547-b0cc-109fbf93ba79-plugins-config-volume" pod "62a84764-3ac8-4547-b0cc-109fbf93ba79" (UID: "62a84764-3ac8-4547-b0cc-109fbf93ba79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:12.103039 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.103015 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f41080-7e48-4aa7-849f-e64d113f21c9-kube-api-access-9b9z2" pod "b6f41080-7e48-4aa7-849f-e64d113f21c9" (UID: "b6f41080-7e48-4aa7-849f-e64d113f21c9"). InnerVolumeSpecName "kube-api-access-9b9z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:12.103144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.103043 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a84764-3ac8-4547-b0cc-109fbf93ba79-kube-api-access-4g4nf" pod "62a84764-3ac8-4547-b0cc-109fbf93ba79" (UID: "62a84764-3ac8-4547-b0cc-109fbf93ba79"). InnerVolumeSpecName "kube-api-access-4g4nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:12.201985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.201951 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f41080-7e48-4aa7-849f-e64d113f21c9-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:12.201985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.201980 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a84764-3ac8-4547-b0cc-109fbf93ba79-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:12.201985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.201991 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9z2\" (UniqueName: \"kubernetes.io/projected/b6f41080-7e48-4aa7-849f-e64d113f21c9-kube-api-access-9b9z2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:12.202241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.202000 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4nf\" (UniqueName: \"kubernetes.io/projected/62a84764-3ac8-4547-b0cc-109fbf93ba79-kube-api-access-4g4nf\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:12.258286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.258246 4948 generic.go:356] "Generic (PLEG): container finished" podID="b6f41080-7e48-4aa7-849f-e64d113f21c9" containerID="733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29" exitCode=0 Nov 03 19:37:12.258286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.258296 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" Nov 03 19:37:12.258568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.258340 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" event={"ID":"b6f41080-7e48-4aa7-849f-e64d113f21c9","Type":"ContainerDied","Data":"733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29"} Nov 03 19:37:12.258568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.258441 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf" event={"ID":"b6f41080-7e48-4aa7-849f-e64d113f21c9","Type":"ContainerDied","Data":"6016134da30176cfe68f9e9e90c7c471c4fd701731667dc8332cbaa6d54cdfdb"} Nov 03 19:37:12.258568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.258474 4948 scope.go:122] "RemoveContainer" containerID="733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29" Nov 03 19:37:12.260140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.260109 4948 generic.go:356] "Generic (PLEG): container finished" podID="62a84764-3ac8-4547-b0cc-109fbf93ba79" containerID="14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0" exitCode=0 Nov 03 19:37:12.260250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.260184 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" event={"ID":"62a84764-3ac8-4547-b0cc-109fbf93ba79","Type":"ContainerDied","Data":"14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0"} Nov 03 19:37:12.260347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.260250 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" event={"ID":"62a84764-3ac8-4547-b0cc-109fbf93ba79","Type":"ContainerDied","Data":"cd24fa7f44788bdd5d1070f11a3372c6e56036a1409f9af8cef61789a6c92000"} Nov 03 19:37:12.260488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.260454 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4" Nov 03 19:37:12.270056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.270033 4948 scope.go:122] "RemoveContainer" containerID="733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29" Nov 03 19:37:12.270443 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:37:12.270414 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29\": container with ID starting with 733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29 not found: ID does not exist" containerID="733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29" Nov 03 19:37:12.270538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.270454 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29"} err="failed to get container status \"733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29\": rpc error: code = NotFound desc = could not find container \"733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29\": container with ID starting with 733bea689e6f9523dcf56e91a43a27df2fe44e5c53f0af6d69b6cffc1e589b29 not found: ID does not exist" Nov 03 19:37:12.270538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.270480 4948 scope.go:122] "RemoveContainer" containerID="14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0" Nov 03 19:37:12.282818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.282792 4948 scope.go:122] "RemoveContainer" containerID="14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0" Nov 03 19:37:12.283125 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:37:12.283102 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0\": container with ID starting with 14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0 not found: ID does not exist" containerID="14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0" Nov 03 19:37:12.283195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.283135 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0"} err="failed to get container status \"14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0\": rpc error: code = NotFound desc = could not find container \"14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0\": container with ID starting with 14c5fd0f67a8c9efa9d84331d47c5e1cdad1e9798c4136b87c4b8e49e841abd0 not found: ID does not exist" Nov 03 19:37:12.427423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.427387 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4"] Nov 03 19:37:12.432934 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.432909 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-020814-6f56b5c9fdkhk4"] Nov 03 19:37:12.545538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.545452 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf"] Nov 03 19:37:12.549045 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.549020 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-020814-67dfbqkbgf"] Nov 03 19:37:12.752304 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.752268 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a84764-3ac8-4547-b0cc-109fbf93ba79" path="/var/lib/kubelet/pods/62a84764-3ac8-4547-b0cc-109fbf93ba79/volumes" Nov 03 19:37:12.752658 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:12.752646 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f41080-7e48-4aa7-849f-e64d113f21c9" path="/var/lib/kubelet/pods/b6f41080-7e48-4aa7-849f-e64d113f21c9/volumes" Nov 03 19:37:30.049237 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:30.049200 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57"] Nov 03 19:37:30.101732 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:30.101678 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9td57"] Nov 03 19:37:30.753062 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:30.753027 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4036196c-3a43-4fe6-8f86-e0eccbdd5843" path="/var/lib/kubelet/pods/4036196c-3a43-4fe6-8f86-e0eccbdd5843/volumes" Nov 03 19:37:32.792616 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:37:32.792574 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:37:34.765067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:34.765031 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr"] Nov 03 19:37:34.765517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:34.765224 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" podUID="ab6a451a-e52a-4951-a1b7-02fa91d55d02" containerName="manager" containerID="cri-o://e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5" gracePeriod=10 Nov 03 19:37:35.024525 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.024463 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 19:37:35.121678 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.121639 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6a451a-e52a-4951-a1b7-02fa91d55d02-kube-api-access-vkd2j\" (UniqueName: \"kubernetes.io/projected/ab6a451a-e52a-4951-a1b7-02fa91d55d02-kube-api-access-vkd2j\") pod \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " Nov 03 19:37:35.121917 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.121843 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6a451a-e52a-4951-a1b7-02fa91d55d02-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab6a451a-e52a-4951-a1b7-02fa91d55d02-extensions-socket-volume\") pod \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\" (UID: \"ab6a451a-e52a-4951-a1b7-02fa91d55d02\") " Nov 03 19:37:35.122209 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.122184 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6a451a-e52a-4951-a1b7-02fa91d55d02-extensions-socket-volume" pod "ab6a451a-e52a-4951-a1b7-02fa91d55d02" (UID: "ab6a451a-e52a-4951-a1b7-02fa91d55d02"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:35.123954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.123929 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6a451a-e52a-4951-a1b7-02fa91d55d02-kube-api-access-vkd2j" pod "ab6a451a-e52a-4951-a1b7-02fa91d55d02" (UID: "ab6a451a-e52a-4951-a1b7-02fa91d55d02"). InnerVolumeSpecName "kube-api-access-vkd2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:35.223385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.223355 4948 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab6a451a-e52a-4951-a1b7-02fa91d55d02-extensions-socket-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:35.223385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.223383 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkd2j\" (UniqueName: \"kubernetes.io/projected/ab6a451a-e52a-4951-a1b7-02fa91d55d02-kube-api-access-vkd2j\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:35.452609 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.452579 4948 generic.go:356] "Generic (PLEG): container finished" podID="ab6a451a-e52a-4951-a1b7-02fa91d55d02" containerID="e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5" exitCode=0 Nov 03 19:37:35.452800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.452631 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" Nov 03 19:37:35.452800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.452673 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" event={"ID":"ab6a451a-e52a-4951-a1b7-02fa91d55d02","Type":"ContainerDied","Data":"e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5"} Nov 03 19:37:35.452800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.452740 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr" event={"ID":"ab6a451a-e52a-4951-a1b7-02fa91d55d02","Type":"ContainerDied","Data":"9c205ae2dad185411fce49793d79baa3e26acf2b7920c59e7490ca1db03d5602"} Nov 03 19:37:35.452800 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.452768 4948 scope.go:122] "RemoveContainer" containerID="e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5" Nov 03 19:37:35.463066 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.463049 4948 scope.go:122] "RemoveContainer" containerID="e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5" Nov 03 19:37:35.463333 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:37:35.463313 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5\": container with ID starting with e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5 not found: ID does not exist" containerID="e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5" Nov 03 19:37:35.463400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.463344 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5"} err="failed to get container status \"e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5\": rpc error: code = NotFound desc = could not find container \"e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5\": container with ID starting with e67ac2b9a5ef70bde559605786ecb10cb87746f081c447ad701ea2abd567cfd5 not found: ID does not exist" Nov 03 19:37:35.621638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.621600 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr"] Nov 03 19:37:35.672414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:35.672379 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brr"] Nov 03 19:37:36.752470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:36.752435 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab6a451a-e52a-4951-a1b7-02fa91d55d02" path="/var/lib/kubelet/pods/ab6a451a-e52a-4951-a1b7-02fa91d55d02/volumes" Nov 03 19:37:55.809743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:55.809694 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:37:55.816097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:55.816076 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:37:56.070385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:56.070291 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:37:56.077237 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:37:56.077217 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:38:01.305263 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:01.305222 4948 scope.go:122] "RemoveContainer" containerID="fd3af3d581c9b1eab2836f4231216123dede835ff43ecf742d58e5cb7d73caa6" Nov 03 19:38:01.314602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:01.314579 4948 scope.go:122] "RemoveContainer" containerID="583338ce681bb2635783b1e6e8078920e1a0b0bc6bd8a3a24e898ab4362465e7" Nov 03 19:38:02.792251 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:38:02.792213 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:38:32.792539 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:38:32.792448 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:38:38.395759 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.395716 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7"] Nov 03 19:38:38.396657 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396631 4948 cpu_manager.go:414] "containerMap: removing container" podUID="ab6a451a-e52a-4951-a1b7-02fa91d55d02" containerName="manager" Nov 03 19:38:38.396812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396659 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab6a451a-e52a-4951-a1b7-02fa91d55d02" containerName="manager" Nov 03 19:38:38.396812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396726 4948 cpu_manager.go:414] "containerMap: removing container" podUID="10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" containerName="revoke-keys" Nov 03 19:38:38.396812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396736 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" containerName="revoke-keys" Nov 03 19:38:38.396812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396758 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" Nov 03 19:38:38.396812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396765 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" Nov 03 19:38:38.396812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396778 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" Nov 03 19:38:38.396812 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396786 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396824 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b6f41080-7e48-4aa7-849f-e64d113f21c9" containerName="payload-pre-processing" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396832 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f41080-7e48-4aa7-849f-e64d113f21c9" containerName="payload-pre-processing" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396856 4948 cpu_manager.go:414] "containerMap: removing container" podUID="62a84764-3ac8-4547-b0cc-109fbf93ba79" containerName="payload-processing" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.396864 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a84764-3ac8-4547-b0cc-109fbf93ba79" containerName="payload-processing" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.397057 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" containerName="revoke-keys" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.397070 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.397078 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c5d9bf-f941-421e-ab76-cdd44dfca7a6" containerName="istio-proxy" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.397094 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f41080-7e48-4aa7-849f-e64d113f21c9" containerName="payload-pre-processing" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.397112 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a84764-3ac8-4547-b0cc-109fbf93ba79" containerName="payload-processing" Nov 03 19:38:38.397167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.397123 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab6a451a-e52a-4951-a1b7-02fa91d55d02" containerName="manager" Nov 03 19:38:38.401922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.401899 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:38.460272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.460237 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7"] Nov 03 19:38:38.507690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.507659 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mfqs9\"" Nov 03 19:38:38.531167 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.531137 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjq6n\" (UniqueName: \"kubernetes.io/projected/d991fd98-c238-4214-b938-daa90d01841f-kube-api-access-cjq6n\") pod \"kuadrant-operator-controller-manager-678d87766f-2q8j7\" (UID: \"d991fd98-c238-4214-b938-daa90d01841f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:38.531294 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.531198 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d991fd98-c238-4214-b938-daa90d01841f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2q8j7\" (UID: \"d991fd98-c238-4214-b938-daa90d01841f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:38.632178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.632143 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjq6n\" (UniqueName: \"kubernetes.io/projected/d991fd98-c238-4214-b938-daa90d01841f-kube-api-access-cjq6n\") pod \"kuadrant-operator-controller-manager-678d87766f-2q8j7\" (UID: \"d991fd98-c238-4214-b938-daa90d01841f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:38.632358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.632229 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d991fd98-c238-4214-b938-daa90d01841f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2q8j7\" (UID: \"d991fd98-c238-4214-b938-daa90d01841f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:38.632621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.632600 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d991fd98-c238-4214-b938-daa90d01841f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2q8j7\" (UID: \"d991fd98-c238-4214-b938-daa90d01841f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:38.700236 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.700204 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjq6n\" (UniqueName: \"kubernetes.io/projected/d991fd98-c238-4214-b938-daa90d01841f-kube-api-access-cjq6n\") pod \"kuadrant-operator-controller-manager-678d87766f-2q8j7\" (UID: \"d991fd98-c238-4214-b938-daa90d01841f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:38.715404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:38.715381 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:39.201175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:39.201149 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7"] Nov 03 19:38:39.202939 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:38:39.202899 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd991fd98_c238_4214_b938_daa90d01841f.slice/crio-8b6ec32f43c2152370e828d63ab5d4062aead8f80ba42cf614419986628835b9 WatchSource:0}: Error finding container 8b6ec32f43c2152370e828d63ab5d4062aead8f80ba42cf614419986628835b9: Status 404 returned error can't find the container with id 8b6ec32f43c2152370e828d63ab5d4062aead8f80ba42cf614419986628835b9 Nov 03 19:38:39.851122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:39.851084 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" event={"ID":"d991fd98-c238-4214-b938-daa90d01841f","Type":"ContainerStarted","Data":"168954c7cfe674c847b20d121a726737e0f9832726df432d9e9ac8415a34d788"} Nov 03 19:38:39.851534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:39.851134 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" event={"ID":"d991fd98-c238-4214-b938-daa90d01841f","Type":"ContainerStarted","Data":"8b6ec32f43c2152370e828d63ab5d4062aead8f80ba42cf614419986628835b9"} Nov 03 19:38:39.851534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:39.851222 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:38:40.025547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:40.025492 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" podStartSLOduration=2.025477033 podStartE2EDuration="2.025477033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:39.972133064 +0000 UTC m=+3649.455128979" watchObservedRunningTime="2026-11-03 19:38:40.025477033 +0000 UTC m=+3649.508472947" Nov 03 19:38:50.860767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:38:50.860735 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2q8j7" Nov 03 19:39:01.373724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:01.371668 4948 scope.go:122] "RemoveContainer" containerID="f79f7adf75a0b430a58c700374cad7e0f3adfa4affac5bdf21591f66217941fa" Nov 03 19:39:01.404085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:01.404038 4948 scope.go:122] "RemoveContainer" containerID="18b69f7285d3e193cf83a5590db134969a0bc8dcec39a9953876bf7bd129b60e" Nov 03 19:39:01.456906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:01.456877 4948 scope.go:122] "RemoveContainer" containerID="b41b7d4f8a47bcd72b57de5141c0cd0aed0bfe79a969fc0e300844805c06ed93" Nov 03 19:39:02.851161 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:39:02.851079 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:39:04.159296 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.159261 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88"] Nov 03 19:39:04.165202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.165176 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.215376 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.215342 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88"] Nov 03 19:39:04.227351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.227322 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkx2\" (UniqueName: \"kubernetes.io/projected/1d609a8b-c373-4a28-a109-cc48d330d2f4-kube-api-access-dbkx2\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.227509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.227359 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.227509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.227408 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.263588 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.263553 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:39:04.263794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.263606 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:39:04.263794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.263640 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:39:04.263794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.263644 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:39:04.328181 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.328140 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.328392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.328257 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkx2\" (UniqueName: \"kubernetes.io/projected/1d609a8b-c373-4a28-a109-cc48d330d2f4-kube-api-access-dbkx2\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.328392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.328288 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.329047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.329019 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.330633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.330610 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.392987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.392949 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkx2\" (UniqueName: \"kubernetes.io/projected/1d609a8b-c373-4a28-a109-cc48d330d2f4-kube-api-access-dbkx2\") pod \"maas-api-revoke-sub-d576e68b-w7j88\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.478962 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.478922 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:04.858788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:04.858751 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88"] Nov 03 19:39:04.859597 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:39:04.859568 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d609a8b_c373_4a28_a109_cc48d330d2f4.slice/crio-425dcd021044334607bd8d4b0712ec65980b6fa6c227fb6d9bceda8b546fca02 WatchSource:0}: Error finding container 425dcd021044334607bd8d4b0712ec65980b6fa6c227fb6d9bceda8b546fca02: Status 404 returned error can't find the container with id 425dcd021044334607bd8d4b0712ec65980b6fa6c227fb6d9bceda8b546fca02 Nov 03 19:39:05.051304 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:05.051275 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp"] Nov 03 19:39:05.054360 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:05.054327 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2tkhp"] Nov 03 19:39:05.056675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:05.056648 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" event={"ID":"1d609a8b-c373-4a28-a109-cc48d330d2f4","Type":"ContainerStarted","Data":"425dcd021044334607bd8d4b0712ec65980b6fa6c227fb6d9bceda8b546fca02"} Nov 03 19:39:06.064203 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:06.064121 4948 generic.go:356] "Generic (PLEG): container finished" podID="1d609a8b-c373-4a28-a109-cc48d330d2f4" containerID="aa70f77fa2353f59cd0580eef14d0bf64c74da548e8107a738db62b75be50bf8" exitCode=0 Nov 03 19:39:06.064567 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:06.064204 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" event={"ID":"1d609a8b-c373-4a28-a109-cc48d330d2f4","Type":"ContainerDied","Data":"aa70f77fa2353f59cd0580eef14d0bf64c74da548e8107a738db62b75be50bf8"} Nov 03 19:39:06.752891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:06.752852 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759975d3-885b-4904-896f-0e2f3867dac3" path="/var/lib/kubelet/pods/759975d3-885b-4904-896f-0e2f3867dac3/volumes" Nov 03 19:39:07.222135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.222110 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:07.256093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.256062 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-openshift-ca\") pod \"1d609a8b-c373-4a28-a109-cc48d330d2f4\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " Nov 03 19:39:07.256282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.256105 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-xks-ca\") pod \"1d609a8b-c373-4a28-a109-cc48d330d2f4\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " Nov 03 19:39:07.256282 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.256155 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d609a8b-c373-4a28-a109-cc48d330d2f4-kube-api-access-dbkx2\" (UniqueName: \"kubernetes.io/projected/1d609a8b-c373-4a28-a109-cc48d330d2f4-kube-api-access-dbkx2\") pod \"1d609a8b-c373-4a28-a109-cc48d330d2f4\" (UID: \"1d609a8b-c373-4a28-a109-cc48d330d2f4\") " Nov 03 19:39:07.256488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.256453 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-openshift-ca" pod "1d609a8b-c373-4a28-a109-cc48d330d2f4" (UID: "1d609a8b-c373-4a28-a109-cc48d330d2f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:39:07.256661 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.256641 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:07.258864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.258835 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-xks-ca" pod "1d609a8b-c373-4a28-a109-cc48d330d2f4" (UID: "1d609a8b-c373-4a28-a109-cc48d330d2f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:39:07.258967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.258867 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d609a8b-c373-4a28-a109-cc48d330d2f4-kube-api-access-dbkx2" pod "1d609a8b-c373-4a28-a109-cc48d330d2f4" (UID: "1d609a8b-c373-4a28-a109-cc48d330d2f4"). InnerVolumeSpecName "kube-api-access-dbkx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:39:07.358284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.358190 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkx2\" (UniqueName: \"kubernetes.io/projected/1d609a8b-c373-4a28-a109-cc48d330d2f4-kube-api-access-dbkx2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:07.358284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:07.358229 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d609a8b-c373-4a28-a109-cc48d330d2f4-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:08.078639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:08.078602 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" Nov 03 19:39:08.078832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:08.078638 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88" event={"ID":"1d609a8b-c373-4a28-a109-cc48d330d2f4","Type":"ContainerDied","Data":"425dcd021044334607bd8d4b0712ec65980b6fa6c227fb6d9bceda8b546fca02"} Nov 03 19:39:08.078832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:08.078685 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="425dcd021044334607bd8d4b0712ec65980b6fa6c227fb6d9bceda8b546fca02" Nov 03 19:39:17.118666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:17.118623 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w"] Nov 03 19:39:17.173401 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:17.169998 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqh7w"] Nov 03 19:39:18.752448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:18.752415 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8933f286-39a0-465f-9b6e-301557aa4157" path="/var/lib/kubelet/pods/8933f286-39a0-465f-9b6e-301557aa4157/volumes" Nov 03 19:39:32.793216 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:39:32.793176 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:39:33.185790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.185755 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh"] Nov 03 19:39:33.186431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.186415 4948 cpu_manager.go:414] "containerMap: removing container" podUID="1d609a8b-c373-4a28-a109-cc48d330d2f4" containerName="revoke-keys" Nov 03 19:39:33.186483 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.186434 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d609a8b-c373-4a28-a109-cc48d330d2f4" containerName="revoke-keys" Nov 03 19:39:33.186608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.186595 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d609a8b-c373-4a28-a109-cc48d330d2f4" containerName="revoke-keys" Nov 03 19:39:33.190328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.190307 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.202476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.202438 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.202640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.202601 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.202640 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.202633 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8qd4\" (UniqueName: \"kubernetes.io/projected/d57e66eb-19c3-40ca-922f-1c06092eef16-kube-api-access-s8qd4\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.241350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.241316 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh"] Nov 03 19:39:33.289553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.289516 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:39:33.289775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.289564 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:39:33.289775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.289569 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:39:33.289775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.289569 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:39:33.303382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.303351 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.303563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.303389 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8qd4\" (UniqueName: \"kubernetes.io/projected/d57e66eb-19c3-40ca-922f-1c06092eef16-kube-api-access-s8qd4\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.303563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.303423 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.304134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.304112 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.305799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.305775 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.365925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.365889 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8qd4\" (UniqueName: \"kubernetes.io/projected/d57e66eb-19c3-40ca-922f-1c06092eef16-kube-api-access-s8qd4\") pod \"maas-api-revoke-sub-d2bc93bc-mc5sh\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.504004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.503916 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:33.889993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:33.889964 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh"] Nov 03 19:39:33.891536 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:39:33.891500 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57e66eb_19c3_40ca_922f_1c06092eef16.slice/crio-09cbda6b7b1c2d144131bdf7aa0c2ec6073ad6a39201d258c1cd204e7866aa07 WatchSource:0}: Error finding container 09cbda6b7b1c2d144131bdf7aa0c2ec6073ad6a39201d258c1cd204e7866aa07: Status 404 returned error can't find the container with id 09cbda6b7b1c2d144131bdf7aa0c2ec6073ad6a39201d258c1cd204e7866aa07 Nov 03 19:39:34.396660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:34.396624 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" event={"ID":"d57e66eb-19c3-40ca-922f-1c06092eef16","Type":"ContainerStarted","Data":"f3903e7627a83f9456b20071040d6ca3649e28f26e42907e85f3be05bc9d38be"} Nov 03 19:39:34.396867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:34.396683 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" event={"ID":"d57e66eb-19c3-40ca-922f-1c06092eef16","Type":"ContainerStarted","Data":"09cbda6b7b1c2d144131bdf7aa0c2ec6073ad6a39201d258c1cd204e7866aa07"} Nov 03 19:39:34.516673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:34.516615 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" podStartSLOduration=1.5165998429999998 podStartE2EDuration="1.516599843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:34.513961684 +0000 UTC m=+3703.996957599" watchObservedRunningTime="2026-11-03 19:39:34.516599843 +0000 UTC m=+3703.999595794" Nov 03 19:39:35.404993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:35.404958 4948 generic.go:356] "Generic (PLEG): container finished" podID="d57e66eb-19c3-40ca-922f-1c06092eef16" containerID="f3903e7627a83f9456b20071040d6ca3649e28f26e42907e85f3be05bc9d38be" exitCode=0 Nov 03 19:39:35.405509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:35.405049 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" event={"ID":"d57e66eb-19c3-40ca-922f-1c06092eef16","Type":"ContainerDied","Data":"f3903e7627a83f9456b20071040d6ca3649e28f26e42907e85f3be05bc9d38be"} Nov 03 19:39:36.548947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.548920 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:36.633889 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.633850 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-openshift-ca\") pod \"d57e66eb-19c3-40ca-922f-1c06092eef16\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " Nov 03 19:39:36.633889 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.633893 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57e66eb-19c3-40ca-922f-1c06092eef16-kube-api-access-s8qd4\" (UniqueName: \"kubernetes.io/projected/d57e66eb-19c3-40ca-922f-1c06092eef16-kube-api-access-s8qd4\") pod \"d57e66eb-19c3-40ca-922f-1c06092eef16\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " Nov 03 19:39:36.634153 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.633929 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-xks-ca\") pod \"d57e66eb-19c3-40ca-922f-1c06092eef16\" (UID: \"d57e66eb-19c3-40ca-922f-1c06092eef16\") " Nov 03 19:39:36.634285 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.634259 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-openshift-ca" pod "d57e66eb-19c3-40ca-922f-1c06092eef16" (UID: "d57e66eb-19c3-40ca-922f-1c06092eef16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:39:36.636125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.636100 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-xks-ca" pod "d57e66eb-19c3-40ca-922f-1c06092eef16" (UID: "d57e66eb-19c3-40ca-922f-1c06092eef16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:39:36.636404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.636380 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57e66eb-19c3-40ca-922f-1c06092eef16-kube-api-access-s8qd4" pod "d57e66eb-19c3-40ca-922f-1c06092eef16" (UID: "d57e66eb-19c3-40ca-922f-1c06092eef16"). InnerVolumeSpecName "kube-api-access-s8qd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:39:36.734723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.734667 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:36.734723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.734694 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8qd4\" (UniqueName: \"kubernetes.io/projected/d57e66eb-19c3-40ca-922f-1c06092eef16-kube-api-access-s8qd4\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:36.734723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:36.734732 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e66eb-19c3-40ca-922f-1c06092eef16-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:37.417886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:37.417854 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" Nov 03 19:39:37.418071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:37.417890 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh" event={"ID":"d57e66eb-19c3-40ca-922f-1c06092eef16","Type":"ContainerDied","Data":"09cbda6b7b1c2d144131bdf7aa0c2ec6073ad6a39201d258c1cd204e7866aa07"} Nov 03 19:39:37.418071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:37.417933 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09cbda6b7b1c2d144131bdf7aa0c2ec6073ad6a39201d258c1cd204e7866aa07" Nov 03 19:39:46.223635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.223591 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8"] Nov 03 19:39:46.224249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.224230 4948 cpu_manager.go:414] "containerMap: removing container" podUID="d57e66eb-19c3-40ca-922f-1c06092eef16" containerName="revoke-keys" Nov 03 19:39:46.224306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.224252 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57e66eb-19c3-40ca-922f-1c06092eef16" containerName="revoke-keys" Nov 03 19:39:46.224428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.224415 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57e66eb-19c3-40ca-922f-1c06092eef16" containerName="revoke-keys" Nov 03 19:39:46.228340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.228317 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.281708 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.281676 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-439a13-openshift-default-dockercfg-n6fwt\"" Nov 03 19:39:46.323162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323132 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-token\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323168 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istiod-ca-cert\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323200 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-podinfo\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323301 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-envoy\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323358 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-data\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323389 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-socket\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323458 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-certs\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323515 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-credential-socket\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.323746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.323627 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmn2\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-kube-api-access-tmmn2\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.335575 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.335545 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8"] Nov 03 19:39:46.424931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.424895 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmn2\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-kube-api-access-tmmn2\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.424943 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-token\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425192 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425172 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istiod-ca-cert\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425205 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-podinfo\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425250 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425223 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-envoy\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425252 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-data\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425280 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-socket\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425329 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-certs\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425369 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-credential-socket\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425719 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425677 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-data\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425729 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-socket\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425796 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-certs\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.425913 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.425860 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-credential-socket\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.426334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.426308 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istiod-ca-cert\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.427819 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.427799 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-envoy\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.428038 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.428019 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-podinfo\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.487935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.487850 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmn2\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-kube-api-access-tmmn2\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.491474 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.491448 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-token\") pod \"e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.549222 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.549177 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:46.996399 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.996365 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8"] Nov 03 19:39:46.997201 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:39:46.997176 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc53bbc1_6026_443e_b5a4_aa7e1a46256c.slice/crio-bd544c09a6ed1c033c83e50a832c204e8cf137b73c37cafcf34b0fa38ca641a1 WatchSource:0}: Error finding container bd544c09a6ed1c033c83e50a832c204e8cf137b73c37cafcf34b0fa38ca641a1: Status 404 returned error can't find the container with id bd544c09a6ed1c033c83e50a832c204e8cf137b73c37cafcf34b0fa38ca641a1 Nov 03 19:39:46.999420 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.999389 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:46.999526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.999452 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:46.999526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:46.999480 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:47.486133 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:47.486099 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" event={"ID":"dc53bbc1-6026-443e-b5a4-aa7e1a46256c","Type":"ContainerStarted","Data":"0b2bf01ae3d06c817e6efebc9ccc11326f77224d4edfca53efaf79bc221e82db"} Nov 03 19:39:47.486629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:47.486159 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" event={"ID":"dc53bbc1-6026-443e-b5a4-aa7e1a46256c","Type":"ContainerStarted","Data":"bd544c09a6ed1c033c83e50a832c204e8cf137b73c37cafcf34b0fa38ca641a1"} Nov 03 19:39:47.549563 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:47.549534 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:47.611465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:47.611410 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" podStartSLOduration=1.6113959169999998 podStartE2EDuration="1.611395917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:47.609269671 +0000 UTC m=+3717.092265586" watchObservedRunningTime="2026-11-03 19:39:47.611395917 +0000 UTC m=+3717.094391831" Nov 03 19:39:48.555589 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:48.555556 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:49.502368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:49.502332 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:49.503267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:49.503243 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:39:54.431947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.431897 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf"] Nov 03 19:39:54.437241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.437205 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.490191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.490160 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf"] Nov 03 19:39:54.506756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.506724 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5fcd877-bf3c-42bf-8f28-f25c393a8090-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.506908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.506876 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqggc\" (UniqueName: \"kubernetes.io/projected/c5fcd877-bf3c-42bf-8f28-f25c393a8090-kube-api-access-nqggc\") pod \"payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.540836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.540799 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0a156c6a-dockercfg-szcqs\"" Nov 03 19:39:54.541062 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.540900 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-439a13\"" Nov 03 19:39:54.578425 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.578393 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx"] Nov 03 19:39:54.582270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.582247 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:54.607885 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.607853 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5fcd877-bf3c-42bf-8f28-f25c393a8090-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.608034 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.607961 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqggc\" (UniqueName: \"kubernetes.io/projected/c5fcd877-bf3c-42bf-8f28-f25c393a8090-kube-api-access-nqggc\") pod \"payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.608558 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.608537 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5fcd877-bf3c-42bf-8f28-f25c393a8090-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.635900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.635869 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx"] Nov 03 19:39:54.669866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.669828 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqggc\" (UniqueName: \"kubernetes.io/projected/c5fcd877-bf3c-42bf-8f28-f25c393a8090-kube-api-access-nqggc\") pod \"payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.708627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.708596 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8tj\" (UniqueName: \"kubernetes.io/projected/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-kube-api-access-qh8tj\") pod \"payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:54.708813 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.708649 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:54.753473 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.753448 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:54.809401 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.809363 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:54.809602 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.809545 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8tj\" (UniqueName: \"kubernetes.io/projected/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-kube-api-access-qh8tj\") pod \"payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:54.810386 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.810360 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:54.871690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.871654 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8tj\" (UniqueName: \"kubernetes.io/projected/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-kube-api-access-qh8tj\") pod \"payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:54.895572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:54.895539 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:55.128198 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.128167 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf"] Nov 03 19:39:55.128963 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:39:55.128936 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fcd877_bf3c_42bf_8f28_f25c393a8090.slice/crio-50e5e3f72b04d51320a4262473a9acafe979fb3ebff9eb764e09c36a1416bd29 WatchSource:0}: Error finding container 50e5e3f72b04d51320a4262473a9acafe979fb3ebff9eb764e09c36a1416bd29: Status 404 returned error can't find the container with id 50e5e3f72b04d51320a4262473a9acafe979fb3ebff9eb764e09c36a1416bd29 Nov 03 19:39:55.271451 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.271418 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx"] Nov 03 19:39:55.272112 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:39:55.272083 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8db2e2_f297_4d17_b55f_a639d4a9ef8f.slice/crio-2fc30ab91b06a3cb763f5202d1d664c72c0d84c336daf6a30665c8717bfeeea2 WatchSource:0}: Error finding container 2fc30ab91b06a3cb763f5202d1d664c72c0d84c336daf6a30665c8717bfeeea2: Status 404 returned error can't find the container with id 2fc30ab91b06a3cb763f5202d1d664c72c0d84c336daf6a30665c8717bfeeea2 Nov 03 19:39:55.555096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.555063 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" event={"ID":"c5fcd877-bf3c-42bf-8f28-f25c393a8090","Type":"ContainerStarted","Data":"9d6295621ab8acee275404d4cc0c12f0f857f056f69343f022d07dfbe3a928f5"} Nov 03 19:39:55.555574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.555119 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" event={"ID":"c5fcd877-bf3c-42bf-8f28-f25c393a8090","Type":"ContainerStarted","Data":"50e5e3f72b04d51320a4262473a9acafe979fb3ebff9eb764e09c36a1416bd29"} Nov 03 19:39:55.555574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.555259 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:39:55.556850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.556825 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" event={"ID":"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f","Type":"ContainerStarted","Data":"1c59ca359da478512b9606a486cc64977d59280481e74caa0e3954615a67dd5d"} Nov 03 19:39:55.557005 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.556868 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" event={"ID":"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f","Type":"ContainerStarted","Data":"2fc30ab91b06a3cb763f5202d1d664c72c0d84c336daf6a30665c8717bfeeea2"} Nov 03 19:39:55.557005 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.556897 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:39:55.675116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:55.675055 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" podStartSLOduration=1.675039916 podStartE2EDuration="1.675039916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:55.673153567 +0000 UTC m=+3725.156149486" watchObservedRunningTime="2026-11-03 19:39:55.675039916 +0000 UTC m=+3725.158035830" Nov 03 19:39:57.151756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.151679 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" podStartSLOduration=3.151663099 podStartE2EDuration="3.151663099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:55.793468782 +0000 UTC m=+3725.276464696" watchObservedRunningTime="2026-11-03 19:39:57.151663099 +0000 UTC m=+3726.634659013" Nov 03 19:39:57.155470 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.155440 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2"] Nov 03 19:39:57.160358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.160335 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.234667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.234639 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.234836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.234676 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-metrics-certs\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.234836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.234785 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d7e97-ce3d-41ed-a35b-2a04fc57a180-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.234836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.234833 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmqlt\" (UniqueName: \"kubernetes.io/projected/604d7e97-ce3d-41ed-a35b-2a04fc57a180-kube-api-access-cmqlt\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.266593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.266544 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:39:57.266593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.266572 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-439a13-serving-cert\"" Nov 03 19:39:57.266593 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.266579 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:39:57.266867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.266633 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\"" Nov 03 19:39:57.266867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.266728 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 19:39:57.269570 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.269540 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2"] Nov 03 19:39:57.335954 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.335919 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d7e97-ce3d-41ed-a35b-2a04fc57a180-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.336164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.336005 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqlt\" (UniqueName: \"kubernetes.io/projected/604d7e97-ce3d-41ed-a35b-2a04fc57a180-kube-api-access-cmqlt\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.336164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.336084 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.336164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.336118 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-metrics-certs\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.336358 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:39:57.336214 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-serving-cert: secret "maas-api-e2e-worker-single-main-439a13-serving-cert" not found Nov 03 19:39:57.336358 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:39:57.336271 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls podName:604d7e97-ce3d-41ed-a35b-2a04fc57a180 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:57.836257817 +0000 UTC m=+3727.319253723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls") pod "maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" (UID: "604d7e97-ce3d-41ed-a35b-2a04fc57a180") : secret "maas-api-e2e-worker-single-main-439a13-serving-cert" not found Nov 03 19:39:57.336616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.336595 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d7e97-ce3d-41ed-a35b-2a04fc57a180-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.338680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.338653 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-metrics-certs\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.411524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.411440 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqlt\" (UniqueName: \"kubernetes.io/projected/604d7e97-ce3d-41ed-a35b-2a04fc57a180-kube-api-access-cmqlt\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.841081 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.841044 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:57.843574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:57.843549 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:58.075963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:58.075926 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:58.454195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:58.454163 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2"] Nov 03 19:39:58.455487 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:39:58.455462 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604d7e97_ce3d_41ed_a35b_2a04fc57a180.slice/crio-c248fdf2d5f49711123029374abaeb6a31d12f0d9a00ed269684e60f6c2dda80 WatchSource:0}: Error finding container c248fdf2d5f49711123029374abaeb6a31d12f0d9a00ed269684e60f6c2dda80: Status 404 returned error can't find the container with id c248fdf2d5f49711123029374abaeb6a31d12f0d9a00ed269684e60f6c2dda80 Nov 03 19:39:58.576511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:58.576469 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" event={"ID":"604d7e97-ce3d-41ed-a35b-2a04fc57a180","Type":"ContainerStarted","Data":"c248fdf2d5f49711123029374abaeb6a31d12f0d9a00ed269684e60f6c2dda80"} Nov 03 19:39:59.583927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:59.583833 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" event={"ID":"604d7e97-ce3d-41ed-a35b-2a04fc57a180","Type":"ContainerStarted","Data":"e5b8fef92f93518a013aa4cde20026a872217830e8fdf27d598ea584306e64ad"} Nov 03 19:39:59.583927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:59.583916 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:39:59.706062 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:39:59.705998 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" podStartSLOduration=1.8410213930000001 podStartE2EDuration="2.705984003s" totalImagesPullingTime="864.96261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:59.7031119 +0000 UTC m=+3729.186107814" watchObservedRunningTime="2026-11-03 19:39:59.705984003 +0000 UTC m=+3729.188979917" Nov 03 19:40:01.617833 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:01.617758 4948 scope.go:122] "RemoveContainer" containerID="674bf4d15a7f77c6233b7ebd4a6114aefea8d28e8c391730f14284a3881894c9" Nov 03 19:40:02.794616 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:40:02.794574 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:40:05.595573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:05.595544 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:40:06.576374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:06.576334 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:40:06.577350 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:06.577321 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:40:15.709859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:15.709820 4948 generic.go:356] "Generic (PLEG): container finished" podID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerID="0b2bf01ae3d06c817e6efebc9ccc11326f77224d4edfca53efaf79bc221e82db" exitCode=0 Nov 03 19:40:15.710344 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:15.709864 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" event={"ID":"dc53bbc1-6026-443e-b5a4-aa7e1a46256c","Type":"ContainerDied","Data":"0b2bf01ae3d06c817e6efebc9ccc11326f77224d4edfca53efaf79bc221e82db"} Nov 03 19:40:15.710426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:15.710409 4948 scope.go:122] "RemoveContainer" containerID="0b2bf01ae3d06c817e6efebc9ccc11326f77224d4edfca53efaf79bc221e82db" Nov 03 19:40:15.711620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:15.711592 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:15.711781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:15.711672 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:15.711781 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:15.711739 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:16.718725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:16.718660 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" event={"ID":"dc53bbc1-6026-443e-b5a4-aa7e1a46256c","Type":"ContainerStarted","Data":"3ed44c50f93bcdf2233a06447f302645ede443254217a9ead5394aad31645d93"} Nov 03 19:40:17.550046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:17.550010 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:40:17.551845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:17.551470 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 19:40:17.551845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:17.551532 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 19:40:18.550559 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:18.550521 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 19:40:18.551115 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:18.550605 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 19:40:19.503100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.503067 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:40:19.550030 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.549996 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 19:40:19.550247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.550066 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 19:40:19.625974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.625943 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6"] Nov 03 19:40:19.629072 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.629049 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.687688 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.687653 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6"] Nov 03 19:40:19.705997 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.705958 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-model-cache\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.706233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.706026 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6156849f-06ee-498c-bcbc-6ab084fc72dc-tls-certs\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.706233 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.706136 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-dshm\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.706364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.706228 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgc77\" (UniqueName: \"kubernetes.io/projected/6156849f-06ee-498c-bcbc-6ab084fc72dc-kube-api-access-hgc77\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.706364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.706301 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-tmp-dir\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.706364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.706357 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-home\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.732691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.732663 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-439a13\"/\"openshift-service-ca.crt\"" Nov 03 19:40:19.732691 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.732664 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-439a13\"/\"kube-root-ca.crt\"" Nov 03 19:40:19.732919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.732738 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-439a13\"/\"simulator-single-main-439a13-kserve-self-signed-certs\"" Nov 03 19:40:19.732919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.732786 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-439a13\"/\"default-dockercfg-tnts8\"" Nov 03 19:40:19.807527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.807427 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgc77\" (UniqueName: \"kubernetes.io/projected/6156849f-06ee-498c-bcbc-6ab084fc72dc-kube-api-access-hgc77\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.807717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.807539 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-tmp-dir\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.807717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.807626 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-home\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.807717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.807674 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-model-cache\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.807937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.807751 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6156849f-06ee-498c-bcbc-6ab084fc72dc-tls-certs\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.807937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.807806 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-dshm\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.808070 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.808031 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-tmp-dir\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.808127 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.808079 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-model-cache\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.808286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.808266 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-home\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.809978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.809954 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-dshm\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.810317 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.810295 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6156849f-06ee-498c-bcbc-6ab084fc72dc-tls-certs\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.878291 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.878256 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgc77\" (UniqueName: \"kubernetes.io/projected/6156849f-06ee-498c-bcbc-6ab084fc72dc-kube-api-access-hgc77\") pod \"simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:19.983929 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:19.983893 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:20.377539 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:40:20.377492 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6156849f_06ee_498c_bcbc_6ab084fc72dc.slice/crio-1ea75258ad03d0addc286091813bead1e14eed145599d7f2eae411de660f344a WatchSource:0}: Error finding container 1ea75258ad03d0addc286091813bead1e14eed145599d7f2eae411de660f344a: Status 404 returned error can't find the container with id 1ea75258ad03d0addc286091813bead1e14eed145599d7f2eae411de660f344a Nov 03 19:40:20.377692 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:20.377649 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6"] Nov 03 19:40:20.549922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:20.549888 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 19:40:20.550097 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:20.549938 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 19:40:20.778712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:20.778654 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" event={"ID":"6156849f-06ee-498c-bcbc-6ab084fc72dc","Type":"ContainerStarted","Data":"367f98c92ce6b9d7edd8af4e6c07fc7de23b96a2a973f4db6400f53903169cb3"} Nov 03 19:40:20.778712 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:20.778717 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" event={"ID":"6156849f-06ee-498c-bcbc-6ab084fc72dc","Type":"ContainerStarted","Data":"1ea75258ad03d0addc286091813bead1e14eed145599d7f2eae411de660f344a"} Nov 03 19:40:20.876925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:20.876867 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" podStartSLOduration=1.876851116 podStartE2EDuration="1.876851116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:20.876072052 +0000 UTC m=+3750.359067969" watchObservedRunningTime="2026-11-03 19:40:20.876851116 +0000 UTC m=+3750.359847029" Nov 03 19:40:20.987772 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:20.984821 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:21.550496 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:21.550454 4948 patch_prober.go:29] interesting pod/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 19:40:21.550667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:21.550526 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 19:40:22.553764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:22.553732 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:40:22.554909 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:22.554881 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:40:29.984219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:29.984187 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:29.996818 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:29.996793 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:30.838866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:40:30.838839 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:40:32.792145 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:40:32.792108 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:41:01.643396 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:01.643360 4948 scope.go:122] "RemoveContainer" containerID="012afdda0725045f6648e2f9eb73b0e45f9451c6ccc21d6d0ebd6487975887bc" Nov 03 19:41:01.652687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:01.652663 4948 scope.go:122] "RemoveContainer" containerID="9387955ad4979db1b4350bd549d5e85f1e07a7d51dc774993a3e32c9223f3263" Nov 03 19:41:01.661367 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:01.661346 4948 scope.go:122] "RemoveContainer" containerID="d79110ad02dcc5c3553cea9f8e901ef6dbf217361b7552975d2112841fb0a10a" Nov 03 19:41:02.792194 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:41:02.792157 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:41:21.090926 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:21.090888 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6"] Nov 03 19:41:27.972051 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:27.972008 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2"] Nov 03 19:41:27.972543 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:27.972240 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" podUID="604d7e97-ce3d-41ed-a35b-2a04fc57a180" containerName="maas-api" containerID="cri-o://e5b8fef92f93518a013aa4cde20026a872217830e8fdf27d598ea584306e64ad" gracePeriod=30 Nov 03 19:41:28.196861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:28.196817 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx"] Nov 03 19:41:28.305170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:28.305070 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf"] Nov 03 19:41:29.204145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.204118 4948 generic.go:356] "Generic (PLEG): container finished" podID="604d7e97-ce3d-41ed-a35b-2a04fc57a180" containerID="e5b8fef92f93518a013aa4cde20026a872217830e8fdf27d598ea584306e64ad" exitCode=0 Nov 03 19:41:29.204481 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.204187 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" event={"ID":"604d7e97-ce3d-41ed-a35b-2a04fc57a180","Type":"ContainerDied","Data":"e5b8fef92f93518a013aa4cde20026a872217830e8fdf27d598ea584306e64ad"} Nov 03 19:41:29.222501 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.222477 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:41:29.232757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.232736 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d7e97-ce3d-41ed-a35b-2a04fc57a180-kube-api-access-cmqlt\" (UniqueName: \"kubernetes.io/projected/604d7e97-ce3d-41ed-a35b-2a04fc57a180-kube-api-access-cmqlt\") pod \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " Nov 03 19:41:29.232893 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.232877 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604d7e97-ce3d-41ed-a35b-2a04fc57a180-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d7e97-ce3d-41ed-a35b-2a04fc57a180-openshift-service-ca\") pod \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " Nov 03 19:41:29.232940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.232920 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls\") pod \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " Nov 03 19:41:29.232985 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.232945 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-metrics-certs\") pod \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\" (UID: \"604d7e97-ce3d-41ed-a35b-2a04fc57a180\") " Nov 03 19:41:29.233255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.233223 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604d7e97-ce3d-41ed-a35b-2a04fc57a180-openshift-service-ca" pod "604d7e97-ce3d-41ed-a35b-2a04fc57a180" (UID: "604d7e97-ce3d-41ed-a35b-2a04fc57a180"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:29.234977 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.234947 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-metrics-certs" pod "604d7e97-ce3d-41ed-a35b-2a04fc57a180" (UID: "604d7e97-ce3d-41ed-a35b-2a04fc57a180"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:29.235085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.235028 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls" pod "604d7e97-ce3d-41ed-a35b-2a04fc57a180" (UID: "604d7e97-ce3d-41ed-a35b-2a04fc57a180"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:29.235085 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.235065 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d7e97-ce3d-41ed-a35b-2a04fc57a180-kube-api-access-cmqlt" pod "604d7e97-ce3d-41ed-a35b-2a04fc57a180" (UID: "604d7e97-ce3d-41ed-a35b-2a04fc57a180"). InnerVolumeSpecName "kube-api-access-cmqlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:29.334083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.334009 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d7e97-ce3d-41ed-a35b-2a04fc57a180-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:29.334083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.334040 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:29.334083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.334049 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d7e97-ce3d-41ed-a35b-2a04fc57a180-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:29.334083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:29.334058 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmqlt\" (UniqueName: \"kubernetes.io/projected/604d7e97-ce3d-41ed-a35b-2a04fc57a180-kube-api-access-cmqlt\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:30.211416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:30.211384 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" Nov 03 19:41:30.211864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:30.211416 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2" event={"ID":"604d7e97-ce3d-41ed-a35b-2a04fc57a180","Type":"ContainerDied","Data":"c248fdf2d5f49711123029374abaeb6a31d12f0d9a00ed269684e60f6c2dda80"} Nov 03 19:41:30.211864 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:30.211476 4948 scope.go:122] "RemoveContainer" containerID="e5b8fef92f93518a013aa4cde20026a872217830e8fdf27d598ea584306e64ad" Nov 03 19:41:30.382723 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:30.381849 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2"] Nov 03 19:41:30.389152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:30.389119 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-439a13-75cfd5cc5c-gngs2"] Nov 03 19:41:30.753146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:30.753075 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604d7e97-ce3d-41ed-a35b-2a04fc57a180" path="/var/lib/kubelet/pods/604d7e97-ce3d-41ed-a35b-2a04fc57a180/volumes" Nov 03 19:41:32.814070 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:41:32.813999 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:41:33.197443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:33.197398 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" podUID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" containerName="payload-processing" containerID="cri-o://1c59ca359da478512b9606a486cc64977d59280481e74caa0e3954615a67dd5d" gracePeriod=25 Nov 03 19:41:33.305427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:33.305367 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" podUID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" containerName="payload-pre-processing" containerID="cri-o://9d6295621ab8acee275404d4cc0c12f0f857f056f69343f022d07dfbe3a928f5" gracePeriod=25 Nov 03 19:41:36.111608 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.111563 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" podUID="6156849f-06ee-498c-bcbc-6ab084fc72dc" containerName="main" containerID="cri-o://367f98c92ce6b9d7edd8af4e6c07fc7de23b96a2a973f4db6400f53903169cb3" gracePeriod=45 Nov 03 19:41:36.251648 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.251616 4948 generic.go:356] "Generic (PLEG): container finished" podID="6156849f-06ee-498c-bcbc-6ab084fc72dc" containerID="367f98c92ce6b9d7edd8af4e6c07fc7de23b96a2a973f4db6400f53903169cb3" exitCode=0 Nov 03 19:41:36.251873 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.251714 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" event={"ID":"6156849f-06ee-498c-bcbc-6ab084fc72dc","Type":"ContainerDied","Data":"367f98c92ce6b9d7edd8af4e6c07fc7de23b96a2a973f4db6400f53903169cb3"} Nov 03 19:41:36.371443 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.371385 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:41:36.401718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.401678 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-dshm\") pod \"6156849f-06ee-498c-bcbc-6ab084fc72dc\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " Nov 03 19:41:36.401847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.401787 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-model-cache\") pod \"6156849f-06ee-498c-bcbc-6ab084fc72dc\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " Nov 03 19:41:36.401847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.401815 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6156849f-06ee-498c-bcbc-6ab084fc72dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/6156849f-06ee-498c-bcbc-6ab084fc72dc-tls-certs\") pod \"6156849f-06ee-498c-bcbc-6ab084fc72dc\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " Nov 03 19:41:36.401958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.401862 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6156849f-06ee-498c-bcbc-6ab084fc72dc-kube-api-access-hgc77\" (UniqueName: \"kubernetes.io/projected/6156849f-06ee-498c-bcbc-6ab084fc72dc-kube-api-access-hgc77\") pod \"6156849f-06ee-498c-bcbc-6ab084fc72dc\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " Nov 03 19:41:36.401958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.401903 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-tmp-dir\") pod \"6156849f-06ee-498c-bcbc-6ab084fc72dc\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " Nov 03 19:41:36.401958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.401928 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-home\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-home\") pod \"6156849f-06ee-498c-bcbc-6ab084fc72dc\" (UID: \"6156849f-06ee-498c-bcbc-6ab084fc72dc\") " Nov 03 19:41:36.402235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.402168 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-model-cache" pod "6156849f-06ee-498c-bcbc-6ab084fc72dc" (UID: "6156849f-06ee-498c-bcbc-6ab084fc72dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:36.402235 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.402197 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-tmp-dir" pod "6156849f-06ee-498c-bcbc-6ab084fc72dc" (UID: "6156849f-06ee-498c-bcbc-6ab084fc72dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:36.402454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.402292 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:36.402454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.402312 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:36.402454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.402400 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-home" pod "6156849f-06ee-498c-bcbc-6ab084fc72dc" (UID: "6156849f-06ee-498c-bcbc-6ab084fc72dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:36.404147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.404116 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6156849f-06ee-498c-bcbc-6ab084fc72dc-tls-certs" pod "6156849f-06ee-498c-bcbc-6ab084fc72dc" (UID: "6156849f-06ee-498c-bcbc-6ab084fc72dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:36.404286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.404198 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6156849f-06ee-498c-bcbc-6ab084fc72dc-kube-api-access-hgc77" pod "6156849f-06ee-498c-bcbc-6ab084fc72dc" (UID: "6156849f-06ee-498c-bcbc-6ab084fc72dc"). InnerVolumeSpecName "kube-api-access-hgc77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:36.404286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.404277 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-dshm" pod "6156849f-06ee-498c-bcbc-6ab084fc72dc" (UID: "6156849f-06ee-498c-bcbc-6ab084fc72dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:36.503635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.503605 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6156849f-06ee-498c-bcbc-6ab084fc72dc-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:36.503635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.503633 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgc77\" (UniqueName: \"kubernetes.io/projected/6156849f-06ee-498c-bcbc-6ab084fc72dc-kube-api-access-hgc77\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:36.503903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.503644 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:36.503903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.503655 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6156849f-06ee-498c-bcbc-6ab084fc72dc-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:36.564875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.564837 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" podUID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:41:36.564875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:36.564850 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" podUID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:41:37.259936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:37.259903 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" Nov 03 19:41:37.260431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:37.259931 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6" event={"ID":"6156849f-06ee-498c-bcbc-6ab084fc72dc","Type":"ContainerDied","Data":"1ea75258ad03d0addc286091813bead1e14eed145599d7f2eae411de660f344a"} Nov 03 19:41:37.260431 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:37.259992 4948 scope.go:122] "RemoveContainer" containerID="367f98c92ce6b9d7edd8af4e6c07fc7de23b96a2a973f4db6400f53903169cb3" Nov 03 19:41:37.427275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:37.427242 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6"] Nov 03 19:41:37.430906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:37.430878 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-439a13/simulator-single-main-439a13-kserve-6fd75fc76f-r5fq6"] Nov 03 19:41:38.757123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:38.757084 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6156849f-06ee-498c-bcbc-6ab084fc72dc" path="/var/lib/kubelet/pods/6156849f-06ee-498c-bcbc-6ab084fc72dc/volumes" Nov 03 19:41:40.193094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:40.193055 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8"] Nov 03 19:41:40.193518 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:40.193262 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" containerID="cri-o://3ed44c50f93bcdf2233a06447f302645ede443254217a9ead5394aad31645d93" gracePeriod=30 Nov 03 19:41:45.325032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.324999 4948 generic.go:356] "Generic (PLEG): container finished" podID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerID="3ed44c50f93bcdf2233a06447f302645ede443254217a9ead5394aad31645d93" exitCode=0 Nov 03 19:41:45.325517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.325068 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" event={"ID":"dc53bbc1-6026-443e-b5a4-aa7e1a46256c","Type":"ContainerDied","Data":"3ed44c50f93bcdf2233a06447f302645ede443254217a9ead5394aad31645d93"} Nov 03 19:41:45.325517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.325130 4948 scope.go:122] "RemoveContainer" containerID="0b2bf01ae3d06c817e6efebc9ccc11326f77224d4edfca53efaf79bc221e82db" Nov 03 19:41:45.459666 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.459609 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:41:45.596144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596111 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-podinfo\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596149 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-certs\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596292 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-kube-api-access-tmmn2\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-kube-api-access-tmmn2\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596328 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-token\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-token\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596421 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-envoy\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596472 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istiod-ca-cert\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596474 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-certs" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:45.596682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596512 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-socket\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596603 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-credential-socket\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596682 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596642 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-data\") pod \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\" (UID: \"dc53bbc1-6026-443e-b5a4-aa7e1a46256c\") " Nov 03 19:41:45.596903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596852 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-credential-socket" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:45.596903 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.596878 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istiod-ca-cert" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:45.597063 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.597037 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-data" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:45.597195 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.597171 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.597265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.597194 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.597265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.597210 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.597265 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.597224 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.597454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.597272 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-socket" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:45.598510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.598488 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-kube-api-access-tmmn2" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "kube-api-access-tmmn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:45.598510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.598495 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-podinfo" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:41:45.598920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.598897 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-envoy" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:45.598994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.598912 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-token" pod "dc53bbc1-6026-443e-b5a4-aa7e1a46256c" (UID: "dc53bbc1-6026-443e-b5a4-aa7e1a46256c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:45.698179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.698149 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.698179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.698176 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmmn2\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-kube-api-access-tmmn2\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.698385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.698188 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.698385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.698198 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:45.698385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:45.698206 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc53bbc1-6026-443e-b5a4-aa7e1a46256c-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:46.332332 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.332297 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" Nov 03 19:41:46.332840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.332333 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8" event={"ID":"dc53bbc1-6026-443e-b5a4-aa7e1a46256c","Type":"ContainerDied","Data":"bd544c09a6ed1c033c83e50a832c204e8cf137b73c37cafcf34b0fa38ca641a1"} Nov 03 19:41:46.332840 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.332396 4948 scope.go:122] "RemoveContainer" containerID="3ed44c50f93bcdf2233a06447f302645ede443254217a9ead5394aad31645d93" Nov 03 19:41:46.551523 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.551488 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8"] Nov 03 19:41:46.555033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.555007 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-439a13-openshift-default-75d865948fmgxp8"] Nov 03 19:41:46.564587 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.564549 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" podUID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:41:46.564742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.564549 4948 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" podUID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:41:46.674721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.674632 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp"] Nov 03 19:41:46.675170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675146 4948 cpu_manager.go:414] "containerMap: removing container" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" Nov 03 19:41:46.675170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675168 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675199 4948 cpu_manager.go:414] "containerMap: removing container" podUID="604d7e97-ce3d-41ed-a35b-2a04fc57a180" containerName="maas-api" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675206 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604d7e97-ce3d-41ed-a35b-2a04fc57a180" containerName="maas-api" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675217 4948 cpu_manager.go:414] "containerMap: removing container" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675222 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675249 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6156849f-06ee-498c-bcbc-6ab084fc72dc" containerName="main" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675255 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6156849f-06ee-498c-bcbc-6ab084fc72dc" containerName="main" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675321 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="6156849f-06ee-498c-bcbc-6ab084fc72dc" containerName="main" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675329 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" Nov 03 19:41:46.675336 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675340 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="604d7e97-ce3d-41ed-a35b-2a04fc57a180" containerName="maas-api" Nov 03 19:41:46.675618 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.675348 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" containerName="istio-proxy" Nov 03 19:41:46.679743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.679725 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.732890 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.732855 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp"] Nov 03 19:41:46.733424 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.733405 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a05bf110-kserve-self-signed-certs\"" Nov 03 19:41:46.753902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.753872 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc53bbc1-6026-443e-b5a4-aa7e1a46256c" path="/var/lib/kubelet/pods/dc53bbc1-6026-443e-b5a4-aa7e1a46256c/volumes" Nov 03 19:41:46.808680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.808623 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-model-cache\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.809071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.809027 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tmp-dir\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.809256 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.809209 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-dshm\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.809446 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.809413 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w28h\" (UniqueName: \"kubernetes.io/projected/f6ec882c-188e-4620-a43a-2b87e8dc7b21-kube-api-access-2w28h\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.809553 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.809501 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tls-certs\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.809642 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.809571 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-home\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911016 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tls-certs\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911064 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-home\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911131 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-model-cache\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911174 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tmp-dir\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911248 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911244 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-dshm\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911304 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w28h\" (UniqueName: \"kubernetes.io/projected/f6ec882c-188e-4620-a43a-2b87e8dc7b21-kube-api-access-2w28h\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911505 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-home\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.911995 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911944 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tmp-dir\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.912137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.911991 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-model-cache\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.913860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.913838 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-dshm\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.913993 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.913953 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tls-certs\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:46.973157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:46.973125 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w28h\" (UniqueName: \"kubernetes.io/projected/f6ec882c-188e-4620-a43a-2b87e8dc7b21-kube-api-access-2w28h\") pod \"e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:47.002025 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:47.001996 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:47.399899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:47.399868 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp"] Nov 03 19:41:47.401155 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:41:47.401127 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ec882c_188e_4620_a43a_2b87e8dc7b21.slice/crio-12a08437057ab359142c6067ca2a7be56b475445da70fa36a30ef5a840ec1d98 WatchSource:0}: Error finding container 12a08437057ab359142c6067ca2a7be56b475445da70fa36a30ef5a840ec1d98: Status 404 returned error can't find the container with id 12a08437057ab359142c6067ca2a7be56b475445da70fa36a30ef5a840ec1d98 Nov 03 19:41:48.347531 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:48.347489 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" event={"ID":"f6ec882c-188e-4620-a43a-2b87e8dc7b21","Type":"ContainerStarted","Data":"6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02"} Nov 03 19:41:48.347725 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:48.347540 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" event={"ID":"f6ec882c-188e-4620-a43a-2b87e8dc7b21","Type":"ContainerStarted","Data":"12a08437057ab359142c6067ca2a7be56b475445da70fa36a30ef5a840ec1d98"} Nov 03 19:41:48.467229 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:48.467178 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" podStartSLOduration=2.467164554 podStartE2EDuration="2.467164554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:48.465602241 +0000 UTC m=+3837.948598154" watchObservedRunningTime="2026-11-03 19:41:48.467164554 +0000 UTC m=+3837.950160467" Nov 03 19:41:49.003012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:49.002980 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:53.395710 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:53.395410 4948 generic.go:356] "Generic (PLEG): container finished" podID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" containerID="1c59ca359da478512b9606a486cc64977d59280481e74caa0e3954615a67dd5d" exitCode=0 Nov 03 19:41:53.395710 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:53.395512 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" event={"ID":"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f","Type":"ContainerDied","Data":"1c59ca359da478512b9606a486cc64977d59280481e74caa0e3954615a67dd5d"} Nov 03 19:41:54.329032 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.328997 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:41:54.338733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.336965 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-kube-api-access-qh8tj\" (UniqueName: \"kubernetes.io/projected/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-kube-api-access-qh8tj\") pod \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " Nov 03 19:41:54.338733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.337122 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-plugins-config-volume\") pod \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\" (UID: \"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f\") " Nov 03 19:41:54.338733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.337838 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-plugins-config-volume" pod "3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" (UID: "3d8db2e2-f297-4d17-b55f-a639d4a9ef8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:54.569363 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.520024 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-kube-api-access-qh8tj" pod "3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" (UID: "3d8db2e2-f297-4d17-b55f-a639d4a9ef8f"). InnerVolumeSpecName "kube-api-access-qh8tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:54.613760 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.567572 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:54.613972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.603063 4948 generic.go:356] "Generic (PLEG): container finished" podID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" containerID="9d6295621ab8acee275404d4cc0c12f0f857f056f69343f022d07dfbe3a928f5" exitCode=0 Nov 03 19:41:54.622850 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.603095 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" event={"ID":"c5fcd877-bf3c-42bf-8f28-f25c393a8090","Type":"ContainerDied","Data":"9d6295621ab8acee275404d4cc0c12f0f857f056f69343f022d07dfbe3a928f5"} Nov 03 19:41:54.640270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.640237 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" event={"ID":"3d8db2e2-f297-4d17-b55f-a639d4a9ef8f","Type":"ContainerDied","Data":"2fc30ab91b06a3cb763f5202d1d664c72c0d84c336daf6a30665c8717bfeeea2"} Nov 03 19:41:54.668718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.667087 4948 scope.go:122] "RemoveContainer" containerID="1c59ca359da478512b9606a486cc64977d59280481e74caa0e3954615a67dd5d" Nov 03 19:41:54.668718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.644825 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx" Nov 03 19:41:54.668718 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.668434 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:41:54.729400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.729370 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh8tj\" (UniqueName: \"kubernetes.io/projected/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f-kube-api-access-qh8tj\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:54.844927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.839286 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fcd877-bf3c-42bf-8f28-f25c393a8090-kube-api-access-nqggc\" (UniqueName: \"kubernetes.io/projected/c5fcd877-bf3c-42bf-8f28-f25c393a8090-kube-api-access-nqggc\") pod \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " Nov 03 19:41:54.844927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.839395 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fcd877-bf3c-42bf-8f28-f25c393a8090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5fcd877-bf3c-42bf-8f28-f25c393a8090-plugins-config-volume\") pod \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\" (UID: \"c5fcd877-bf3c-42bf-8f28-f25c393a8090\") " Nov 03 19:41:54.844927 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.843923 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fcd877-bf3c-42bf-8f28-f25c393a8090-plugins-config-volume" pod "c5fcd877-bf3c-42bf-8f28-f25c393a8090" (UID: "c5fcd877-bf3c-42bf-8f28-f25c393a8090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:54.881822 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.872362 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fcd877-bf3c-42bf-8f28-f25c393a8090-kube-api-access-nqggc" pod "c5fcd877-bf3c-42bf-8f28-f25c393a8090" (UID: "c5fcd877-bf3c-42bf-8f28-f25c393a8090"). InnerVolumeSpecName "kube-api-access-nqggc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:54.940218 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.940191 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqggc\" (UniqueName: \"kubernetes.io/projected/c5fcd877-bf3c-42bf-8f28-f25c393a8090-kube-api-access-nqggc\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:54.940429 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.940416 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5fcd877-bf3c-42bf-8f28-f25c393a8090-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:55.004600 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:54.998190 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx"] Nov 03 19:41:55.009673 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:55.009646 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-439a13-7d55dcff4925rx"] Nov 03 19:41:55.647916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:55.647838 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" Nov 03 19:41:55.648364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:55.647834 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf" event={"ID":"c5fcd877-bf3c-42bf-8f28-f25c393a8090","Type":"ContainerDied","Data":"50e5e3f72b04d51320a4262473a9acafe979fb3ebff9eb764e09c36a1416bd29"} Nov 03 19:41:55.648364 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:55.648028 4948 scope.go:122] "RemoveContainer" containerID="9d6295621ab8acee275404d4cc0c12f0f857f056f69343f022d07dfbe3a928f5" Nov 03 19:41:55.815669 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:55.815635 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf"] Nov 03 19:41:55.866674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:55.866641 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-439a13-95c6dnrtkf"] Nov 03 19:41:56.753251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:56.753212 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" path="/var/lib/kubelet/pods/3d8db2e2-f297-4d17-b55f-a639d4a9ef8f/volumes" Nov 03 19:41:56.753654 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:56.753638 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" path="/var/lib/kubelet/pods/c5fcd877-bf3c-42bf-8f28-f25c393a8090/volumes" Nov 03 19:41:57.002991 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:57.002953 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:57.015196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:57.015133 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:41:57.676414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:41:57.676386 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:42:02.797808 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:42:02.797765 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:42:04.049743 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:04.049692 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w"] Nov 03 19:42:04.100230 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:04.100196 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m964w"] Nov 03 19:42:04.752341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:04.752306 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bcbc46-49a4-4492-b3f5-a1246e9ad4ea" path="/var/lib/kubelet/pods/10bcbc46-49a4-4492-b3f5-a1246e9ad4ea/volumes" Nov 03 19:42:30.606094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.606058 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76"] Nov 03 19:42:30.606683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.606592 4948 cpu_manager.go:414] "containerMap: removing container" podUID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" containerName="payload-pre-processing" Nov 03 19:42:30.606683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.606614 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" containerName="payload-pre-processing" Nov 03 19:42:30.606683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.606634 4948 cpu_manager.go:414] "containerMap: removing container" podUID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" containerName="payload-processing" Nov 03 19:42:30.606683 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.606643 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" containerName="payload-processing" Nov 03 19:42:30.606865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.606737 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fcd877-bf3c-42bf-8f28-f25c393a8090" containerName="payload-pre-processing" Nov 03 19:42:30.606865 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.606747 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8db2e2-f297-4d17-b55f-a639d4a9ef8f" containerName="payload-processing" Nov 03 19:42:30.610059 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.610038 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.666286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.666252 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76"] Nov 03 19:42:30.713173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.713136 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a05bf110-kserve-self-signed-certs\"" Nov 03 19:42:30.789464 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.789421 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppkb\" (UniqueName: \"kubernetes.io/projected/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-kube-api-access-6ppkb\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.789645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.789474 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tls-certs\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.789645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.789568 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tmp-dir\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.789645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.789622 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-home\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.789691 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-model-cache\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.789792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.789728 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-dshm\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.890511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.890414 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-model-cache\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.890511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.890463 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-dshm\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.890511 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.890508 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppkb\" (UniqueName: \"kubernetes.io/projected/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-kube-api-access-6ppkb\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.890902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.890576 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tls-certs\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.890902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.890629 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tmp-dir\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.890902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.890685 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-home\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.890902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.890887 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-model-cache\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.891134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.891067 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tmp-dir\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.891134 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.891077 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-home\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.892785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.892767 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-dshm\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.893049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.893031 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tls-certs\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:30.955019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:30.954988 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppkb\" (UniqueName: \"kubernetes.io/projected/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-kube-api-access-6ppkb\") pod \"e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:31.221791 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:31.221755 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:31.596627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:31.596601 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76"] Nov 03 19:42:31.597712 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:42:31.597672 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54d3b9f9_3a01_43a7_bc2c_168b464a79bf.slice/crio-e05025f1e936f73100bf3f3a36fa5e4bfc54d7cf99d1380142f90949088d4493 WatchSource:0}: Error finding container e05025f1e936f73100bf3f3a36fa5e4bfc54d7cf99d1380142f90949088d4493: Status 404 returned error can't find the container with id e05025f1e936f73100bf3f3a36fa5e4bfc54d7cf99d1380142f90949088d4493 Nov 03 19:42:31.874971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:31.874876 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" event={"ID":"54d3b9f9-3a01-43a7-bc2c-168b464a79bf","Type":"ContainerStarted","Data":"e317b4af32e6b679b97e0b8c3e65a1b81a00f7fd2bbdd9844b1684666b9940d7"} Nov 03 19:42:31.874971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:31.874928 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" event={"ID":"54d3b9f9-3a01-43a7-bc2c-168b464a79bf","Type":"ContainerStarted","Data":"e05025f1e936f73100bf3f3a36fa5e4bfc54d7cf99d1380142f90949088d4493"} Nov 03 19:42:31.994384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:31.994333 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" podStartSLOduration=1.994318227 podStartE2EDuration="1.994318227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:31.993388493 +0000 UTC m=+3881.476384404" watchObservedRunningTime="2026-11-03 19:42:31.994318227 +0000 UTC m=+3881.477314142" Nov 03 19:42:32.222613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:32.222581 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:32.793435 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:42:32.793390 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:42:41.222402 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:41.222360 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:41.235231 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:41.235201 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:42.027152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:42.027121 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:42:55.856347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:55.856313 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:42:55.862794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:55.862773 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:42:56.115971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:56.115890 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:42:56.122438 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:42:56.122412 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:43:01.728527 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:01.728447 4948 scope.go:122] "RemoveContainer" containerID="4cc0be300ec33b683505f31c5afa73abc04b2e67686aafa491f078b9c2e0a2a3" Nov 03 19:43:02.794108 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:43:02.794063 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:43:03.773670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.773634 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr"] Nov 03 19:43:03.779221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.779201 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.831425 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.831397 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr"] Nov 03 19:43:03.878785 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.878753 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.878931 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.878791 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prnq\" (UniqueName: \"kubernetes.io/projected/19253e67-a331-4aa8-a63e-90fd04d92567-kube-api-access-5prnq\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.878981 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.878952 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.879377 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.879354 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:43:03.879467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.879388 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:43:03.879467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.879404 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:43:03.879467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.879404 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:43:03.980187 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.980156 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.980357 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.980221 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.980357 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.980241 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prnq\" (UniqueName: \"kubernetes.io/projected/19253e67-a331-4aa8-a63e-90fd04d92567-kube-api-access-5prnq\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.981026 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.980997 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:03.982565 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:03.982536 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:04.042754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:04.042664 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prnq\" (UniqueName: \"kubernetes.io/projected/19253e67-a331-4aa8-a63e-90fd04d92567-kube-api-access-5prnq\") pod \"maas-api-revoke-sub-452b6d7b-kktlr\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:04.092453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:04.092423 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:04.467803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:04.467770 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr"] Nov 03 19:43:04.470418 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:43:04.470390 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19253e67_a331_4aa8_a63e_90fd04d92567.slice/crio-4c9c5cc25dee8b6cf133c7d483298e060f0aeb8f0ffc41bd6c239fcedf698980 WatchSource:0}: Error finding container 4c9c5cc25dee8b6cf133c7d483298e060f0aeb8f0ffc41bd6c239fcedf698980: Status 404 returned error can't find the container with id 4c9c5cc25dee8b6cf133c7d483298e060f0aeb8f0ffc41bd6c239fcedf698980 Nov 03 19:43:05.152321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:05.152229 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" event={"ID":"19253e67-a331-4aa8-a63e-90fd04d92567","Type":"ContainerStarted","Data":"46f40ed5aa388694729aff57167165c383f83cd57ce20d8d5f3e6488319a3676"} Nov 03 19:43:05.152321 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:05.152288 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" event={"ID":"19253e67-a331-4aa8-a63e-90fd04d92567","Type":"ContainerStarted","Data":"4c9c5cc25dee8b6cf133c7d483298e060f0aeb8f0ffc41bd6c239fcedf698980"} Nov 03 19:43:05.272409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:05.272346 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" podStartSLOduration=2.272330703 podStartE2EDuration="2.272330703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:05.269882996 +0000 UTC m=+3914.752878911" watchObservedRunningTime="2026-11-03 19:43:05.272330703 +0000 UTC m=+3914.755326618" Nov 03 19:43:06.164132 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:06.164089 4948 generic.go:356] "Generic (PLEG): container finished" podID="19253e67-a331-4aa8-a63e-90fd04d92567" containerID="46f40ed5aa388694729aff57167165c383f83cd57ce20d8d5f3e6488319a3676" exitCode=0 Nov 03 19:43:06.164667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:06.164254 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" event={"ID":"19253e67-a331-4aa8-a63e-90fd04d92567","Type":"ContainerDied","Data":"46f40ed5aa388694729aff57167165c383f83cd57ce20d8d5f3e6488319a3676"} Nov 03 19:43:08.337635 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.337602 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:08.502228 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.502134 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-openshift-ca\") pod \"19253e67-a331-4aa8-a63e-90fd04d92567\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " Nov 03 19:43:08.508169 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.508141 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19253e67-a331-4aa8-a63e-90fd04d92567-kube-api-access-5prnq\" (UniqueName: \"kubernetes.io/projected/19253e67-a331-4aa8-a63e-90fd04d92567-kube-api-access-5prnq\") pod \"19253e67-a331-4aa8-a63e-90fd04d92567\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " Nov 03 19:43:08.517916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.507974 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-openshift-ca" pod "19253e67-a331-4aa8-a63e-90fd04d92567" (UID: "19253e67-a331-4aa8-a63e-90fd04d92567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:08.522775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.517777 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-xks-ca\") pod \"19253e67-a331-4aa8-a63e-90fd04d92567\" (UID: \"19253e67-a331-4aa8-a63e-90fd04d92567\") " Nov 03 19:43:08.532746 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.529164 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:08.568249 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.568213 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19253e67-a331-4aa8-a63e-90fd04d92567-kube-api-access-5prnq" pod "19253e67-a331-4aa8-a63e-90fd04d92567" (UID: "19253e67-a331-4aa8-a63e-90fd04d92567"). InnerVolumeSpecName "kube-api-access-5prnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:08.658722 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.637397 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5prnq\" (UniqueName: \"kubernetes.io/projected/19253e67-a331-4aa8-a63e-90fd04d92567-kube-api-access-5prnq\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:08.661086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.661054 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-xks-ca" pod "19253e67-a331-4aa8-a63e-90fd04d92567" (UID: "19253e67-a331-4aa8-a63e-90fd04d92567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:08.743724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:08.739349 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19253e67-a331-4aa8-a63e-90fd04d92567-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:09.298176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:09.298137 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" event={"ID":"19253e67-a331-4aa8-a63e-90fd04d92567","Type":"ContainerDied","Data":"4c9c5cc25dee8b6cf133c7d483298e060f0aeb8f0ffc41bd6c239fcedf698980"} Nov 03 19:43:09.298502 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:09.298481 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c9c5cc25dee8b6cf133c7d483298e060f0aeb8f0ffc41bd6c239fcedf698980" Nov 03 19:43:09.298721 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:09.298690 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr" Nov 03 19:43:16.678320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:16.678287 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp"] Nov 03 19:43:19.379639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:19.379603 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76"] Nov 03 19:43:24.956325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:24.956287 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4"] Nov 03 19:43:24.956867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:24.956804 4948 cpu_manager.go:414] "containerMap: removing container" podUID="19253e67-a331-4aa8-a63e-90fd04d92567" containerName="revoke-keys" Nov 03 19:43:24.956867 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:24.956820 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19253e67-a331-4aa8-a63e-90fd04d92567" containerName="revoke-keys" Nov 03 19:43:24.957017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:24.956900 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="19253e67-a331-4aa8-a63e-90fd04d92567" containerName="revoke-keys" Nov 03 19:43:24.960306 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:24.960287 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.021878 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.021845 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:43:25.022110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.021874 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:43:25.022110 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.022088 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:43:25.072975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.072939 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:43:25.073894 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.073867 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229r6\" (UniqueName: \"kubernetes.io/projected/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-kube-api-access-229r6\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.074023 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.074007 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.074082 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.074060 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.076049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.076025 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4"] Nov 03 19:43:25.175400 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.175368 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229r6\" (UniqueName: \"kubernetes.io/projected/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-kube-api-access-229r6\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.175592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.175428 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.175592 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.175460 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.176338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.176311 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.178136 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.178113 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.237345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.237267 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229r6\" (UniqueName: \"kubernetes.io/projected/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-kube-api-access-229r6\") pod \"maas-api-revoke-sub-3e31a506-gzjp4\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.274313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.274285 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:25.658362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:25.658336 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4"] Nov 03 19:43:25.660750 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:43:25.660721 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9cbda0_2906_4f82_9a4c_a0ccc1395652.slice/crio-443c012198c0eb83f688cccf4504d1895974e80eedf24dde03886ddb156d903c WatchSource:0}: Error finding container 443c012198c0eb83f688cccf4504d1895974e80eedf24dde03886ddb156d903c: Status 404 returned error can't find the container with id 443c012198c0eb83f688cccf4504d1895974e80eedf24dde03886ddb156d903c Nov 03 19:43:26.458613 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:26.458573 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" event={"ID":"eb9cbda0-2906-4f82-9a4c-a0ccc1395652","Type":"ContainerStarted","Data":"2a2660c4c431efd39070e1a9e2417e9f0ffeb515c1be80396e013a51f9310d4d"} Nov 03 19:43:26.459144 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:26.458637 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" event={"ID":"eb9cbda0-2906-4f82-9a4c-a0ccc1395652","Type":"ContainerStarted","Data":"443c012198c0eb83f688cccf4504d1895974e80eedf24dde03886ddb156d903c"} Nov 03 19:43:26.578277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:26.578192 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" podStartSLOduration=2.578177848 podStartE2EDuration="2.578177848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:26.575623142 +0000 UTC m=+3936.058619056" watchObservedRunningTime="2026-11-03 19:43:26.578177848 +0000 UTC m=+3936.061173762" Nov 03 19:43:27.469808 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:27.469769 4948 generic.go:356] "Generic (PLEG): container finished" podID="eb9cbda0-2906-4f82-9a4c-a0ccc1395652" containerID="2a2660c4c431efd39070e1a9e2417e9f0ffeb515c1be80396e013a51f9310d4d" exitCode=0 Nov 03 19:43:27.470270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:27.469824 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" event={"ID":"eb9cbda0-2906-4f82-9a4c-a0ccc1395652","Type":"ContainerDied","Data":"2a2660c4c431efd39070e1a9e2417e9f0ffeb515c1be80396e013a51f9310d4d"} Nov 03 19:43:28.684281 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.684244 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:28.816120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.816025 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-kube-api-access-229r6\" (UniqueName: \"kubernetes.io/projected/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-kube-api-access-229r6\") pod \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " Nov 03 19:43:28.816847 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.816822 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-xks-ca\") pod \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " Nov 03 19:43:28.817409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.817389 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-openshift-ca\") pod \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\" (UID: \"eb9cbda0-2906-4f82-9a4c-a0ccc1395652\") " Nov 03 19:43:28.818583 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.818558 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-openshift-ca" pod "eb9cbda0-2906-4f82-9a4c-a0ccc1395652" (UID: "eb9cbda0-2906-4f82-9a4c-a0ccc1395652"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:28.820010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.819987 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-kube-api-access-229r6" pod "eb9cbda0-2906-4f82-9a4c-a0ccc1395652" (UID: "eb9cbda0-2906-4f82-9a4c-a0ccc1395652"). InnerVolumeSpecName "kube-api-access-229r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:28.823175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.823140 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-xks-ca" pod "eb9cbda0-2906-4f82-9a4c-a0ccc1395652" (UID: "eb9cbda0-2906-4f82-9a4c-a0ccc1395652"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:28.919251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.919200 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:28.919251 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.919245 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:28.919526 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:28.919262 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229r6\" (UniqueName: \"kubernetes.io/projected/eb9cbda0-2906-4f82-9a4c-a0ccc1395652-kube-api-access-229r6\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:29.505724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:29.505195 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" event={"ID":"eb9cbda0-2906-4f82-9a4c-a0ccc1395652","Type":"ContainerDied","Data":"443c012198c0eb83f688cccf4504d1895974e80eedf24dde03886ddb156d903c"} Nov 03 19:43:29.505724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:29.505260 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="443c012198c0eb83f688cccf4504d1895974e80eedf24dde03886ddb156d903c" Nov 03 19:43:29.505724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:29.505333 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4" Nov 03 19:43:31.698216 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:31.698175 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" podUID="f6ec882c-188e-4620-a43a-2b87e8dc7b21" containerName="main" containerID="cri-o://6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02" gracePeriod=45 Nov 03 19:43:31.954807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:31.954751 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:43:32.019796 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.019758 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ec882c-188e-4620-a43a-2b87e8dc7b21-kube-api-access-2w28h\" (UniqueName: \"kubernetes.io/projected/f6ec882c-188e-4620-a43a-2b87e8dc7b21-kube-api-access-2w28h\") pod \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " Nov 03 19:43:32.019983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.019823 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-model-cache\") pod \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " Nov 03 19:43:32.019983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.019924 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-dshm\") pod \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " Nov 03 19:43:32.020118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.019981 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-home\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-home\") pod \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " Nov 03 19:43:32.020118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020012 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tmp-dir\") pod \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " Nov 03 19:43:32.020118 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020042 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tls-certs\") pod \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\" (UID: \"f6ec882c-188e-4620-a43a-2b87e8dc7b21\") " Nov 03 19:43:32.020268 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020161 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-model-cache" pod "f6ec882c-188e-4620-a43a-2b87e8dc7b21" (UID: "f6ec882c-188e-4620-a43a-2b87e8dc7b21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:32.020331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020283 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-home" pod "f6ec882c-188e-4620-a43a-2b87e8dc7b21" (UID: "f6ec882c-188e-4620-a43a-2b87e8dc7b21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:32.020331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020306 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tmp-dir" pod "f6ec882c-188e-4620-a43a-2b87e8dc7b21" (UID: "f6ec882c-188e-4620-a43a-2b87e8dc7b21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:32.020413 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020395 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:32.020469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020412 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:32.020469 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.020426 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:32.022244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.022217 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ec882c-188e-4620-a43a-2b87e8dc7b21-kube-api-access-2w28h" pod "f6ec882c-188e-4620-a43a-2b87e8dc7b21" (UID: "f6ec882c-188e-4620-a43a-2b87e8dc7b21"). InnerVolumeSpecName "kube-api-access-2w28h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:32.022494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.022470 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-dshm" pod "f6ec882c-188e-4620-a43a-2b87e8dc7b21" (UID: "f6ec882c-188e-4620-a43a-2b87e8dc7b21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:32.022494 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.022467 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tls-certs" pod "f6ec882c-188e-4620-a43a-2b87e8dc7b21" (UID: "f6ec882c-188e-4620-a43a-2b87e8dc7b21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:32.121155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.121123 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ec882c-188e-4620-a43a-2b87e8dc7b21-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:32.121155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.121151 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w28h\" (UniqueName: \"kubernetes.io/projected/f6ec882c-188e-4620-a43a-2b87e8dc7b21-kube-api-access-2w28h\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:32.121155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.121161 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ec882c-188e-4620-a43a-2b87e8dc7b21-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:32.547906 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.547873 4948 generic.go:356] "Generic (PLEG): container finished" podID="f6ec882c-188e-4620-a43a-2b87e8dc7b21" containerID="6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02" exitCode=0 Nov 03 19:43:32.548120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.547930 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" Nov 03 19:43:32.548120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.547955 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" event={"ID":"f6ec882c-188e-4620-a43a-2b87e8dc7b21","Type":"ContainerDied","Data":"6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02"} Nov 03 19:43:32.548120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.548005 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp" event={"ID":"f6ec882c-188e-4620-a43a-2b87e8dc7b21","Type":"ContainerDied","Data":"12a08437057ab359142c6067ca2a7be56b475445da70fa36a30ef5a840ec1d98"} Nov 03 19:43:32.548120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.548028 4948 scope.go:122] "RemoveContainer" containerID="6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02" Nov 03 19:43:32.558338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.558315 4948 scope.go:122] "RemoveContainer" containerID="6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02" Nov 03 19:43:32.558635 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:43:32.558614 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02\": container with ID starting with 6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02 not found: ID does not exist" containerID="6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02" Nov 03 19:43:32.558733 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.558646 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02"} err="failed to get container status \"6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02\": rpc error: code = NotFound desc = could not find container \"6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02\": container with ID starting with 6e8da00cdccb6a845fabf4edf08df68ceb06a3271df103998bf42b9d96adde02 not found: ID does not exist" Nov 03 19:43:32.716889 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.716856 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp"] Nov 03 19:43:32.761861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:32.761828 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a05bf110-kserve-69cb87585-h7nrp"] Nov 03 19:43:32.793944 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:43:32.793908 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:43:34.399330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.399282 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" podUID="54d3b9f9-3a01-43a7-bc2c-168b464a79bf" containerName="main" containerID="cri-o://e317b4af32e6b679b97e0b8c3e65a1b81a00f7fd2bbdd9844b1684666b9940d7" gracePeriod=45 Nov 03 19:43:34.564196 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.564164 4948 generic.go:356] "Generic (PLEG): container finished" podID="54d3b9f9-3a01-43a7-bc2c-168b464a79bf" containerID="e317b4af32e6b679b97e0b8c3e65a1b81a00f7fd2bbdd9844b1684666b9940d7" exitCode=0 Nov 03 19:43:34.564362 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.564243 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" event={"ID":"54d3b9f9-3a01-43a7-bc2c-168b464a79bf","Type":"ContainerDied","Data":"e317b4af32e6b679b97e0b8c3e65a1b81a00f7fd2bbdd9844b1684666b9940d7"} Nov 03 19:43:34.655877 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.655820 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:43:34.744054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744028 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tls-certs\") pod \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " Nov 03 19:43:34.744241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744078 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-home\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-home\") pod \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " Nov 03 19:43:34.744241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744191 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tmp-dir\") pod \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " Nov 03 19:43:34.744241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744225 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-model-cache\") pod \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " Nov 03 19:43:34.744428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744249 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-kube-api-access-6ppkb\" (UniqueName: \"kubernetes.io/projected/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-kube-api-access-6ppkb\") pod \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " Nov 03 19:43:34.744428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744275 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-dshm\") pod \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\" (UID: \"54d3b9f9-3a01-43a7-bc2c-168b464a79bf\") " Nov 03 19:43:34.744428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744371 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-home" pod "54d3b9f9-3a01-43a7-bc2c-168b464a79bf" (UID: "54d3b9f9-3a01-43a7-bc2c-168b464a79bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:34.744605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744482 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tmp-dir" pod "54d3b9f9-3a01-43a7-bc2c-168b464a79bf" (UID: "54d3b9f9-3a01-43a7-bc2c-168b464a79bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:34.744605 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744500 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-model-cache" pod "54d3b9f9-3a01-43a7-bc2c-168b464a79bf" (UID: "54d3b9f9-3a01-43a7-bc2c-168b464a79bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:34.744750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744644 4948 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tmp-dir\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:34.744750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744658 4948 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-model-cache\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:34.744750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.744669 4948 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-home\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:34.746303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.746277 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tls-certs" pod "54d3b9f9-3a01-43a7-bc2c-168b464a79bf" (UID: "54d3b9f9-3a01-43a7-bc2c-168b464a79bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:34.746437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.746340 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-kube-api-access-6ppkb" pod "54d3b9f9-3a01-43a7-bc2c-168b464a79bf" (UID: "54d3b9f9-3a01-43a7-bc2c-168b464a79bf"). InnerVolumeSpecName "kube-api-access-6ppkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:34.746437 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.746362 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-dshm" pod "54d3b9f9-3a01-43a7-bc2c-168b464a79bf" (UID: "54d3b9f9-3a01-43a7-bc2c-168b464a79bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:43:34.752408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.752287 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ec882c-188e-4620-a43a-2b87e8dc7b21" path="/var/lib/kubelet/pods/f6ec882c-188e-4620-a43a-2b87e8dc7b21/volumes" Nov 03 19:43:34.845173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.845139 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ppkb\" (UniqueName: \"kubernetes.io/projected/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-kube-api-access-6ppkb\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:34.845173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.845169 4948 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-dshm\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:34.845173 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:34.845178 4948 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d3b9f9-3a01-43a7-bc2c-168b464a79bf-tls-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:35.572001 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:35.571968 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" Nov 03 19:43:35.572548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:35.571996 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76" event={"ID":"54d3b9f9-3a01-43a7-bc2c-168b464a79bf","Type":"ContainerDied","Data":"e05025f1e936f73100bf3f3a36fa5e4bfc54d7cf99d1380142f90949088d4493"} Nov 03 19:43:35.572548 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:35.572062 4948 scope.go:122] "RemoveContainer" containerID="e317b4af32e6b679b97e0b8c3e65a1b81a00f7fd2bbdd9844b1684666b9940d7" Nov 03 19:43:35.742508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:35.742462 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76"] Nov 03 19:43:35.793480 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:35.793447 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a05bf110-kserve-5c8686b77-42p76"] Nov 03 19:43:36.752936 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:36.752893 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d3b9f9-3a01-43a7-bc2c-168b464a79bf" path="/var/lib/kubelet/pods/54d3b9f9-3a01-43a7-bc2c-168b464a79bf/volumes" Nov 03 19:43:54.537135 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537096 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm"] Nov 03 19:43:54.537596 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537580 4948 cpu_manager.go:414] "containerMap: removing container" podUID="eb9cbda0-2906-4f82-9a4c-a0ccc1395652" containerName="revoke-keys" Nov 03 19:43:54.537643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537597 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb9cbda0-2906-4f82-9a4c-a0ccc1395652" containerName="revoke-keys" Nov 03 19:43:54.537643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537618 4948 cpu_manager.go:414] "containerMap: removing container" podUID="f6ec882c-188e-4620-a43a-2b87e8dc7b21" containerName="main" Nov 03 19:43:54.537643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537623 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ec882c-188e-4620-a43a-2b87e8dc7b21" containerName="main" Nov 03 19:43:54.537643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537638 4948 cpu_manager.go:414] "containerMap: removing container" podUID="54d3b9f9-3a01-43a7-bc2c-168b464a79bf" containerName="main" Nov 03 19:43:54.537643 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537642 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d3b9f9-3a01-43a7-bc2c-168b464a79bf" containerName="main" Nov 03 19:43:54.537826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537718 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d3b9f9-3a01-43a7-bc2c-168b464a79bf" containerName="main" Nov 03 19:43:54.537826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537731 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb9cbda0-2906-4f82-9a4c-a0ccc1395652" containerName="revoke-keys" Nov 03 19:43:54.537826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.537739 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ec882c-188e-4620-a43a-2b87e8dc7b21" containerName="main" Nov 03 19:43:54.542517 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.542495 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.596419 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.596379 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm"] Nov 03 19:43:54.643488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.643452 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:43:54.643488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.643485 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:43:54.643799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.643553 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:43:54.643799 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.643569 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:43:54.652146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.652115 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.652146 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.652148 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.652359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.652170 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9h2p\" (UniqueName: \"kubernetes.io/projected/25498b48-7144-4193-8ddb-2cb0959f330c-kube-api-access-h9h2p\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.753077 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.753047 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.753244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.753085 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.753244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.753105 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h2p\" (UniqueName: \"kubernetes.io/projected/25498b48-7144-4193-8ddb-2cb0959f330c-kube-api-access-h9h2p\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.753740 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.753716 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.755409 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.755388 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.814748 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.814672 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h2p\" (UniqueName: \"kubernetes.io/projected/25498b48-7144-4193-8ddb-2cb0959f330c-kube-api-access-h9h2p\") pod \"maas-api-revoke-sub-d261134f-pdqgm\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:54.855573 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:54.855546 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:55.229638 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:55.229606 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm"] Nov 03 19:43:55.230765 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:43:55.230732 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25498b48_7144_4193_8ddb_2cb0959f330c.slice/crio-3331fbd15054930c910c37a2a7bcfc818ac6b7bfe003ec17a0c984754f08bcee WatchSource:0}: Error finding container 3331fbd15054930c910c37a2a7bcfc818ac6b7bfe003ec17a0c984754f08bcee: Status 404 returned error can't find the container with id 3331fbd15054930c910c37a2a7bcfc818ac6b7bfe003ec17a0c984754f08bcee Nov 03 19:43:55.783810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:55.783772 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" event={"ID":"25498b48-7144-4193-8ddb-2cb0959f330c","Type":"ContainerStarted","Data":"9489082e7fd7340a5b9a1cc956e60b9b7e18a586adfe34c9819bbaad3ec25a68"} Nov 03 19:43:55.784278 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:55.783827 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" event={"ID":"25498b48-7144-4193-8ddb-2cb0959f330c","Type":"ContainerStarted","Data":"3331fbd15054930c910c37a2a7bcfc818ac6b7bfe003ec17a0c984754f08bcee"} Nov 03 19:43:55.902147 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:55.902057 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" podStartSLOduration=1.902045007 podStartE2EDuration="1.902045007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:55.900901228 +0000 UTC m=+3965.383897144" watchObservedRunningTime="2026-11-03 19:43:55.902045007 +0000 UTC m=+3965.385040920" Nov 03 19:43:56.791178 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:56.791142 4948 generic.go:356] "Generic (PLEG): container finished" podID="25498b48-7144-4193-8ddb-2cb0959f330c" containerID="9489082e7fd7340a5b9a1cc956e60b9b7e18a586adfe34c9819bbaad3ec25a68" exitCode=0 Nov 03 19:43:56.791574 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:56.791264 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" event={"ID":"25498b48-7144-4193-8ddb-2cb0959f330c","Type":"ContainerDied","Data":"9489082e7fd7340a5b9a1cc956e60b9b7e18a586adfe34c9819bbaad3ec25a68"} Nov 03 19:43:57.932171 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:57.932148 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:58.082319 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.082248 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-xks-ca\") pod \"25498b48-7144-4193-8ddb-2cb0959f330c\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " Nov 03 19:43:58.082472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.082330 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25498b48-7144-4193-8ddb-2cb0959f330c-kube-api-access-h9h2p\" (UniqueName: \"kubernetes.io/projected/25498b48-7144-4193-8ddb-2cb0959f330c-kube-api-access-h9h2p\") pod \"25498b48-7144-4193-8ddb-2cb0959f330c\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " Nov 03 19:43:58.082472 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.082370 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-openshift-ca\") pod \"25498b48-7144-4193-8ddb-2cb0959f330c\" (UID: \"25498b48-7144-4193-8ddb-2cb0959f330c\") " Nov 03 19:43:58.082892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.082863 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-openshift-ca" pod "25498b48-7144-4193-8ddb-2cb0959f330c" (UID: "25498b48-7144-4193-8ddb-2cb0959f330c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:58.084549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.084524 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-xks-ca" pod "25498b48-7144-4193-8ddb-2cb0959f330c" (UID: "25498b48-7144-4193-8ddb-2cb0959f330c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:58.084549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.084533 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25498b48-7144-4193-8ddb-2cb0959f330c-kube-api-access-h9h2p" pod "25498b48-7144-4193-8ddb-2cb0959f330c" (UID: "25498b48-7144-4193-8ddb-2cb0959f330c"). InnerVolumeSpecName "kube-api-access-h9h2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:58.183476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.183438 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-xks-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:58.183476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.183473 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9h2p\" (UniqueName: \"kubernetes.io/projected/25498b48-7144-4193-8ddb-2cb0959f330c-kube-api-access-h9h2p\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:58.183659 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.183491 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25498b48-7144-4193-8ddb-2cb0959f330c-maas-api-openshift-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:58.803922 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.803893 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" Nov 03 19:43:58.804087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.803923 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm" event={"ID":"25498b48-7144-4193-8ddb-2cb0959f330c","Type":"ContainerDied","Data":"3331fbd15054930c910c37a2a7bcfc818ac6b7bfe003ec17a0c984754f08bcee"} Nov 03 19:43:58.804087 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:43:58.803974 4948 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3331fbd15054930c910c37a2a7bcfc818ac6b7bfe003ec17a0c984754f08bcee" Nov 03 19:44:02.791414 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:44:02.791378 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:44:07.854351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.854315 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-t5x55"] Nov 03 19:44:07.855092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.855061 4948 cpu_manager.go:414] "containerMap: removing container" podUID="25498b48-7144-4193-8ddb-2cb0959f330c" containerName="revoke-keys" Nov 03 19:44:07.855092 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.855084 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25498b48-7144-4193-8ddb-2cb0959f330c" containerName="revoke-keys" Nov 03 19:44:07.855255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.855204 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="25498b48-7144-4193-8ddb-2cb0959f330c" containerName="revoke-keys" Nov 03 19:44:07.859928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.859907 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.874967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.874937 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf"] Nov 03 19:44:07.878866 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.878841 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.889950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.889923 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a80863-a264-4d18-8507-ddc2926a7261-openshift-service-ca\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.890073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.889960 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kxh\" (UniqueName: \"kubernetes.io/projected/6172d739-1b58-45d4-90ae-964dd9491064-kube-api-access-66kxh\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.890073 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.889989 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwknn\" (UniqueName: \"kubernetes.io/projected/87a80863-a264-4d18-8507-ddc2926a7261-kube-api-access-vwknn\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.890193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.890088 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-maas-api-tls\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.890193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.890162 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-metrics-certs\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.890308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.890213 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a80863-a264-4d18-8507-ddc2926a7261-metrics-certs\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.890308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.890233 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a80863-a264-4d18-8507-ddc2926a7261-maas-api-tls\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.890308 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.890264 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172d739-1b58-45d4-90ae-964dd9491064-openshift-service-ca\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.914021 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.913991 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-t5x55"] Nov 03 19:44:07.921404 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.921371 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:44:07.921629 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.921607 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:44:07.962005 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.961969 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\"" Nov 03 19:44:07.962005 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.961978 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 19:44:07.962220 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.962065 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 19:44:07.969140 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.969107 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf"] Nov 03 19:44:07.990892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.990857 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a80863-a264-4d18-8507-ddc2926a7261-openshift-service-ca\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.991054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.990899 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kxh\" (UniqueName: \"kubernetes.io/projected/6172d739-1b58-45d4-90ae-964dd9491064-kube-api-access-66kxh\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.991054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.990934 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwknn\" (UniqueName: \"kubernetes.io/projected/87a80863-a264-4d18-8507-ddc2926a7261-kube-api-access-vwknn\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.991054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.990960 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-maas-api-tls\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.991054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.991005 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-metrics-certs\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.991054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.991036 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a80863-a264-4d18-8507-ddc2926a7261-metrics-certs\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.991337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.991063 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a80863-a264-4d18-8507-ddc2926a7261-maas-api-tls\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.991337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.991103 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172d739-1b58-45d4-90ae-964dd9491064-openshift-service-ca\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.991815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.991790 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a80863-a264-4d18-8507-ddc2926a7261-openshift-service-ca\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.991973 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.991926 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172d739-1b58-45d4-90ae-964dd9491064-openshift-service-ca\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.993628 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.993604 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a80863-a264-4d18-8507-ddc2926a7261-maas-api-tls\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.993788 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.993627 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-maas-api-tls\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:07.993948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.993929 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a80863-a264-4d18-8507-ddc2926a7261-metrics-certs\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:07.994046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:07.994025 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-metrics-certs\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:08.049632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.049599 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88"] Nov 03 19:44:08.053795 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.053770 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w7j88"] Nov 03 19:44:08.054552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.054515 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwknn\" (UniqueName: \"kubernetes.io/projected/87a80863-a264-4d18-8507-ddc2926a7261-kube-api-access-vwknn\") pod \"maas-api-6696c7b56d-kpmhf\" (UID: \"87a80863-a264-4d18-8507-ddc2926a7261\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:08.054815 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.054789 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kxh\" (UniqueName: \"kubernetes.io/projected/6172d739-1b58-45d4-90ae-964dd9491064-kube-api-access-66kxh\") pod \"maas-api-54b47b974d-t5x55\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:08.176154 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.176058 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:08.192427 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.192399 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:08.576061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.576029 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-t5x55"] Nov 03 19:44:08.593948 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:44:08.593917 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6172d739_1b58_45d4_90ae_964dd9491064.slice/crio-1a6e6dc0ac7b76746ee434fcf4ff3106471fdc4c86519d84e15d6e81ab8dd37b WatchSource:0}: Error finding container 1a6e6dc0ac7b76746ee434fcf4ff3106471fdc4c86519d84e15d6e81ab8dd37b: Status 404 returned error can't find the container with id 1a6e6dc0ac7b76746ee434fcf4ff3106471fdc4c86519d84e15d6e81ab8dd37b Nov 03 19:44:08.595244 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.595225 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:44:08.607046 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.607018 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf"] Nov 03 19:44:08.607307 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:44:08.607284 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a80863_a264_4d18_8507_ddc2926a7261.slice/crio-6acd3cf0788c3c17673947c3344a4263e1a4f28095ac266619c334a729e771d8 WatchSource:0}: Error finding container 6acd3cf0788c3c17673947c3344a4263e1a4f28095ac266619c334a729e771d8: Status 404 returned error can't find the container with id 6acd3cf0788c3c17673947c3344a4263e1a4f28095ac266619c334a729e771d8 Nov 03 19:44:08.752932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.752897 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d609a8b-c373-4a28-a109-cc48d330d2f4" path="/var/lib/kubelet/pods/1d609a8b-c373-4a28-a109-cc48d330d2f4/volumes" Nov 03 19:44:08.864061 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.863965 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" event={"ID":"87a80863-a264-4d18-8507-ddc2926a7261","Type":"ContainerStarted","Data":"6acd3cf0788c3c17673947c3344a4263e1a4f28095ac266619c334a729e771d8"} Nov 03 19:44:08.865392 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:08.865367 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" event={"ID":"6172d739-1b58-45d4-90ae-964dd9491064","Type":"ContainerStarted","Data":"1a6e6dc0ac7b76746ee434fcf4ff3106471fdc4c86519d84e15d6e81ab8dd37b"} Nov 03 19:44:09.874033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:09.873992 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" event={"ID":"87a80863-a264-4d18-8507-ddc2926a7261","Type":"ContainerStarted","Data":"fd3700938f88dd2542e8d32c47647361fba03eddbcfcaeb565f42c99f34001fa"} Nov 03 19:44:09.874520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:09.874067 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:09.875911 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:09.875882 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" event={"ID":"6172d739-1b58-45d4-90ae-964dd9491064","Type":"ContainerStarted","Data":"a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2"} Nov 03 19:44:09.876060 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:09.875936 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:09.994807 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:09.994687 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" podStartSLOduration=1.910078721 podStartE2EDuration="2.99467375s" totalImagesPullingTime="1.084595029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:09.992444099 +0000 UTC m=+3979.475440012" watchObservedRunningTime="2026-11-03 19:44:09.99467375 +0000 UTC m=+3979.477669665" Nov 03 19:44:10.114047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:10.113992 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" podStartSLOduration=2.279295893 podStartE2EDuration="3.113977892s" totalImagesPullingTime="834.681999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:10.111373045 +0000 UTC m=+3979.594368960" watchObservedRunningTime="2026-11-03 19:44:10.113977892 +0000 UTC m=+3979.596973805" Nov 03 19:44:14.800937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.800900 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf"] Nov 03 19:44:14.805048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.805025 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.849541 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849494 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-envoy\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.849823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849557 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpq6\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-kube-api-access-vjpq6\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.849823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849658 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-data\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.849823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849689 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istiod-ca-cert\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.849823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849802 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-socket\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.850139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849837 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-credential-socket\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.850139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849878 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-certs\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.850139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.849943 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-podinfo\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.850139 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.850012 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-token\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.858754 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.858728 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-50babf0d-openshift-default-dockercfg-6wjrw\"" Nov 03 19:44:14.913331 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.913296 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf"] Nov 03 19:44:14.950660 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.950612 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-socket\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.963125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.963088 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-credential-socket\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.963351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.963334 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-certs\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.963509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.963491 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-podinfo\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.963684 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.963656 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-token\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.963884 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.963869 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-envoy\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.964064 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.964035 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpq6\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-kube-api-access-vjpq6\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.964241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.964213 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-data\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.964370 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.964354 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istiod-ca-cert\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.965363 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.965335 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istiod-ca-cert\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.965750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.965643 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-credential-socket\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.967408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.966811 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-certs\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.969225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.969157 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-data\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.969734 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.969661 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-socket\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.975680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.975650 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-podinfo\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:14.976674 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:14.976651 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-envoy\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:15.032572 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.032533 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-token\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:15.032731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.032686 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpq6\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-kube-api-access-vjpq6\") pod \"e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:15.118365 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.118282 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:15.557102 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.557069 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf"] Nov 03 19:44:15.558024 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:44:15.557991 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda506cae3_49d9_4b6d_a9da_e8ef55369c7f.slice/crio-b65abc5fddc91c30a315419177baab5ad02c96b3470aa802f61ac823b3242bc0 WatchSource:0}: Error finding container b65abc5fddc91c30a315419177baab5ad02c96b3470aa802f61ac823b3242bc0: Status 404 returned error can't find the container with id b65abc5fddc91c30a315419177baab5ad02c96b3470aa802f61ac823b3242bc0 Nov 03 19:44:15.560287 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.560255 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:15.560358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.560320 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:15.560358 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.560348 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:15.888225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.888149 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-kpmhf" Nov 03 19:44:15.888792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.888772 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:15.915982 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.915948 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" event={"ID":"a506cae3-49d9-4b6d-a9da-e8ef55369c7f","Type":"ContainerStarted","Data":"e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91"} Nov 03 19:44:15.916127 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:15.916000 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" event={"ID":"a506cae3-49d9-4b6d-a9da-e8ef55369c7f","Type":"ContainerStarted","Data":"b65abc5fddc91c30a315419177baab5ad02c96b3470aa802f61ac823b3242bc0"} Nov 03 19:44:16.040340 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.040297 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-t5x55"] Nov 03 19:44:16.040569 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.040545 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" podUID="6172d739-1b58-45d4-90ae-964dd9491064" containerName="maas-api" containerID="cri-o://a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2" gracePeriod=30 Nov 03 19:44:16.119467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.119435 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:16.124476 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.124449 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:16.134687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.134632 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" podStartSLOduration=2.134618999 podStartE2EDuration="2.134618999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:16.133285071 +0000 UTC m=+3985.616280995" watchObservedRunningTime="2026-11-03 19:44:16.134618999 +0000 UTC m=+3985.617614912" Nov 03 19:44:16.325372 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.325349 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:16.377560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.377530 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6172d739-1b58-45d4-90ae-964dd9491064-kube-api-access-66kxh\" (UniqueName: \"kubernetes.io/projected/6172d739-1b58-45d4-90ae-964dd9491064-kube-api-access-66kxh\") pod \"6172d739-1b58-45d4-90ae-964dd9491064\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " Nov 03 19:44:16.377780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.377578 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-maas-api-tls\") pod \"6172d739-1b58-45d4-90ae-964dd9491064\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " Nov 03 19:44:16.377780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.377620 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-metrics-certs\") pod \"6172d739-1b58-45d4-90ae-964dd9491064\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " Nov 03 19:44:16.377780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.377663 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6172d739-1b58-45d4-90ae-964dd9491064-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172d739-1b58-45d4-90ae-964dd9491064-openshift-service-ca\") pod \"6172d739-1b58-45d4-90ae-964dd9491064\" (UID: \"6172d739-1b58-45d4-90ae-964dd9491064\") " Nov 03 19:44:16.378185 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.378155 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6172d739-1b58-45d4-90ae-964dd9491064-openshift-service-ca" pod "6172d739-1b58-45d4-90ae-964dd9491064" (UID: "6172d739-1b58-45d4-90ae-964dd9491064"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:16.379779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.379754 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-maas-api-tls" pod "6172d739-1b58-45d4-90ae-964dd9491064" (UID: "6172d739-1b58-45d4-90ae-964dd9491064"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:16.379881 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.379781 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-metrics-certs" pod "6172d739-1b58-45d4-90ae-964dd9491064" (UID: "6172d739-1b58-45d4-90ae-964dd9491064"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:16.379960 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.379892 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6172d739-1b58-45d4-90ae-964dd9491064-kube-api-access-66kxh" pod "6172d739-1b58-45d4-90ae-964dd9491064" (UID: "6172d739-1b58-45d4-90ae-964dd9491064"). InnerVolumeSpecName "kube-api-access-66kxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:16.479303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.479266 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66kxh\" (UniqueName: \"kubernetes.io/projected/6172d739-1b58-45d4-90ae-964dd9491064-kube-api-access-66kxh\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:16.479303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.479299 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:16.479513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.479316 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6172d739-1b58-45d4-90ae-964dd9491064-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:16.479513 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.479331 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172d739-1b58-45d4-90ae-964dd9491064-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:16.923767 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.923728 4948 generic.go:356] "Generic (PLEG): container finished" podID="6172d739-1b58-45d4-90ae-964dd9491064" containerID="a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2" exitCode=0 Nov 03 19:44:16.924270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.923773 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" Nov 03 19:44:16.924270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.923791 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" event={"ID":"6172d739-1b58-45d4-90ae-964dd9491064","Type":"ContainerDied","Data":"a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2"} Nov 03 19:44:16.924270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.923844 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-t5x55" event={"ID":"6172d739-1b58-45d4-90ae-964dd9491064","Type":"ContainerDied","Data":"1a6e6dc0ac7b76746ee434fcf4ff3106471fdc4c86519d84e15d6e81ab8dd37b"} Nov 03 19:44:16.924270 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.923869 4948 scope.go:122] "RemoveContainer" containerID="a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2" Nov 03 19:44:16.924491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.924467 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:16.925313 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.925292 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:16.934676 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.934659 4948 scope.go:122] "RemoveContainer" containerID="a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2" Nov 03 19:44:16.934964 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:44:16.934945 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2\": container with ID starting with a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2 not found: ID does not exist" containerID="a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2" Nov 03 19:44:16.935022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:16.934971 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2"} err="failed to get container status \"a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2\": rpc error: code = NotFound desc = could not find container \"a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2\": container with ID starting with a686dfcc2b241d12e01d1eaaa72ef0d0ffd9e7b1dd8e0288fd6968d617c2f7e2 not found: ID does not exist" Nov 03 19:44:17.217145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:17.217110 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-t5x55"] Nov 03 19:44:17.220937 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:17.220906 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-t5x55"] Nov 03 19:44:18.754478 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:18.754440 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6172d739-1b58-45d4-90ae-964dd9491064" path="/var/lib/kubelet/pods/6172d739-1b58-45d4-90ae-964dd9491064/volumes" Nov 03 19:44:22.165616 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.165546 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z"] Nov 03 19:44:22.167120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.167090 4948 cpu_manager.go:414] "containerMap: removing container" podUID="6172d739-1b58-45d4-90ae-964dd9491064" containerName="maas-api" Nov 03 19:44:22.167120 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.167119 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6172d739-1b58-45d4-90ae-964dd9491064" containerName="maas-api" Nov 03 19:44:22.167326 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.167216 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="6172d739-1b58-45d4-90ae-964dd9491064" containerName="maas-api" Nov 03 19:44:22.184264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.182146 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.221768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.221733 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z"] Nov 03 19:44:22.235946 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.235912 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8a866-c4df-4295-b700-fd2d7a1de211-plugins-config-volume\") pod \"payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.236193 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.236168 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74t95\" (UniqueName: \"kubernetes.io/projected/cdd8a866-c4df-4295-b700-fd2d7a1de211-kube-api-access-74t95\") pod \"payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.269778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.269746 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-50babf0d\"" Nov 03 19:44:22.269924 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.269851 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-50babf0d-dockercfg-nnv2t\"" Nov 03 19:44:22.337547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.337506 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74t95\" (UniqueName: \"kubernetes.io/projected/cdd8a866-c4df-4295-b700-fd2d7a1de211-kube-api-access-74t95\") pod \"payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.337784 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.337562 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8a866-c4df-4295-b700-fd2d7a1de211-plugins-config-volume\") pod \"payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.338353 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.338328 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8a866-c4df-4295-b700-fd2d7a1de211-plugins-config-volume\") pod \"payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.348255 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.348229 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d"] Nov 03 19:44:22.351961 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.351946 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.399971 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.399940 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74t95\" (UniqueName: \"kubernetes.io/projected/cdd8a866-c4df-4295-b700-fd2d7a1de211-kube-api-access-74t95\") pod \"payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.407099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.407067 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d"] Nov 03 19:44:22.438860 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.438817 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.439028 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.438939 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9vf\" (UniqueName: \"kubernetes.io/projected/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-kube-api-access-xz9vf\") pod \"payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.500004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.499969 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:22.540424 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.540383 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9vf\" (UniqueName: \"kubernetes.io/projected/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-kube-api-access-xz9vf\") pod \"payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.540633 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.540484 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.541215 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.541190 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.602614 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.602584 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9vf\" (UniqueName: \"kubernetes.io/projected/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-kube-api-access-xz9vf\") pod \"payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.665037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.665003 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:22.879122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.879089 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z"] Nov 03 19:44:22.879816 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:44:22.879789 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd8a866_c4df_4295_b700_fd2d7a1de211.slice/crio-da4e43f14c80ea229f8d64ff27169d0efc6c9261f7277493f3091f1ef02d0a01 WatchSource:0}: Error finding container da4e43f14c80ea229f8d64ff27169d0efc6c9261f7277493f3091f1ef02d0a01: Status 404 returned error can't find the container with id da4e43f14c80ea229f8d64ff27169d0efc6c9261f7277493f3091f1ef02d0a01 Nov 03 19:44:22.962327 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.962287 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" event={"ID":"cdd8a866-c4df-4295-b700-fd2d7a1de211","Type":"ContainerStarted","Data":"31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87"} Nov 03 19:44:22.962521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.962345 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" event={"ID":"cdd8a866-c4df-4295-b700-fd2d7a1de211","Type":"ContainerStarted","Data":"da4e43f14c80ea229f8d64ff27169d0efc6c9261f7277493f3091f1ef02d0a01"} Nov 03 19:44:22.962521 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:22.962373 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:23.041090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.041065 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d"] Nov 03 19:44:23.042423 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:44:23.042393 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bf6be9_2b0c_4b58_8bd1_d9e1d7e66551.slice/crio-fbb57b8d594a90c0999a80109c9381860362d5a85f43da1f190bcccc39a50aba WatchSource:0}: Error finding container fbb57b8d594a90c0999a80109c9381860362d5a85f43da1f190bcccc39a50aba: Status 404 returned error can't find the container with id fbb57b8d594a90c0999a80109c9381860362d5a85f43da1f190bcccc39a50aba Nov 03 19:44:23.081519 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.081457 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" podStartSLOduration=1.08144342 podStartE2EDuration="1.08144342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:23.079608853 +0000 UTC m=+3992.562604778" watchObservedRunningTime="2026-11-03 19:44:23.08144342 +0000 UTC m=+3992.564439334" Nov 03 19:44:23.727453 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.727417 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf"] Nov 03 19:44:23.731181 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.731163 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.784465 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.784431 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-50babf0d-serving-cert\"" Nov 03 19:44:23.786794 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.786763 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf"] Nov 03 19:44:23.853056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.853022 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-metrics-certs\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.853056 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.853058 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnv25\" (UniqueName: \"kubernetes.io/projected/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-kube-api-access-cnv25\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.853288 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.853085 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-openshift-service-ca\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.853288 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.853261 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.954158 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.954117 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.954347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.954199 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-metrics-certs\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.954347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.954218 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnv25\" (UniqueName: \"kubernetes.io/projected/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-kube-api-access-cnv25\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.954347 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.954248 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-openshift-service-ca\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.954347 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:44:23.954288 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-serving-cert: secret "maas-api-e2e-ait-50babf0d-serving-cert" not found Nov 03 19:44:23.954577 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:44:23.954376 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls podName:023059f7-e50d-4f01-b7ee-3a1d87c9f0a8 nodeName:}" failed. No retries permitted until 2026-11-03 19:44:24.454354082 +0000 UTC m=+3993.937349991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls") pod "maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" (UID: "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8") : secret "maas-api-e2e-ait-50babf0d-serving-cert" not found Nov 03 19:44:23.954994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.954964 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-openshift-service-ca\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.956590 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.956573 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-metrics-certs\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:23.970766 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.970729 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" event={"ID":"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551","Type":"ContainerStarted","Data":"296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81"} Nov 03 19:44:23.970909 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.970784 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" event={"ID":"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551","Type":"ContainerStarted","Data":"fbb57b8d594a90c0999a80109c9381860362d5a85f43da1f190bcccc39a50aba"} Nov 03 19:44:23.970909 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:23.970810 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:24.016859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:24.016779 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnv25\" (UniqueName: \"kubernetes.io/projected/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-kube-api-access-cnv25\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:24.089407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:24.089355 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" podStartSLOduration=2.089338843 podStartE2EDuration="2.089338843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:24.087364884 +0000 UTC m=+3993.570360799" watchObservedRunningTime="2026-11-03 19:44:24.089338843 +0000 UTC m=+3993.572334759" Nov 03 19:44:24.459622 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:24.459581 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:24.462471 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:24.462438 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls\") pod \"maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:24.644715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:24.644672 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:25.024380 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:25.024347 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf"] Nov 03 19:44:25.024894 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:44:25.024863 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023059f7_e50d_4f01_b7ee_3a1d87c9f0a8.slice/crio-a26a25954fe4b36cd18fb8c7184cee730a03dc25a1a4cdc5e1bbc2423d041fcd WatchSource:0}: Error finding container a26a25954fe4b36cd18fb8c7184cee730a03dc25a1a4cdc5e1bbc2423d041fcd: Status 404 returned error can't find the container with id a26a25954fe4b36cd18fb8c7184cee730a03dc25a1a4cdc5e1bbc2423d041fcd Nov 03 19:44:25.985116 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:25.985073 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" event={"ID":"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8","Type":"ContainerStarted","Data":"18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc"} Nov 03 19:44:25.985324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:25.985139 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" event={"ID":"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8","Type":"ContainerStarted","Data":"a26a25954fe4b36cd18fb8c7184cee730a03dc25a1a4cdc5e1bbc2423d041fcd"} Nov 03 19:44:25.985324 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:25.985269 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:26.107162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:26.107059 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" podStartSLOduration=2.336531154 podStartE2EDuration="3.107042866s" totalImagesPullingTime="770.511712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:26.104080732 +0000 UTC m=+3995.587076656" watchObservedRunningTime="2026-11-03 19:44:26.107042866 +0000 UTC m=+3995.590038825" Nov 03 19:44:31.996824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:31.996743 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:44:32.795672 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:44:32.795629 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:44:33.977933 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:33.977898 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:44:34.981247 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:34.981221 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:44:37.050439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:37.050403 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh"] Nov 03 19:44:37.101351 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:37.101316 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mc5sh"] Nov 03 19:44:38.755181 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:38.755143 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57e66eb-19c3-40ca-922f-1c06092eef16" path="/var/lib/kubelet/pods/d57e66eb-19c3-40ca-922f-1c06092eef16/volumes" Nov 03 19:44:42.120806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:42.120690 4948 generic.go:356] "Generic (PLEG): container finished" podID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerID="e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91" exitCode=0 Nov 03 19:44:42.120806 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:42.120730 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" event={"ID":"a506cae3-49d9-4b6d-a9da-e8ef55369c7f","Type":"ContainerDied","Data":"e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91"} Nov 03 19:44:42.121355 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:42.121141 4948 scope.go:122] "RemoveContainer" containerID="e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91" Nov 03 19:44:42.122576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:42.122543 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:42.122656 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:42.122626 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:42.122717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:42.122657 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:43.129921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:43.129887 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" event={"ID":"a506cae3-49d9-4b6d-a9da-e8ef55369c7f","Type":"ContainerStarted","Data":"dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df"} Nov 03 19:44:44.118653 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:44.118621 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:44.120052 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:44.120027 4948 patch_prober.go:29] interesting pod/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 19:44:44.120145 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:44.120081 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 19:44:45.118855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:45.118817 4948 patch_prober.go:29] interesting pod/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 19:44:45.119267 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:45.118883 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 19:44:46.118768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:46.118728 4948 patch_prober.go:29] interesting pod/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 19:44:46.118987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:46.118781 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 19:44:46.924902 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:46.924865 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:47.119516 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:47.119483 4948 patch_prober.go:29] interesting pod/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 19:44:47.119963 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:47.119535 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 19:44:48.119182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:48.119137 4948 patch_prober.go:29] interesting pod/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 19:44:48.119389 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:48.119220 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 19:44:49.122919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:49.122889 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:44:49.123892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:44:49.123873 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:45:02.793485 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:02.793440 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:45:06.322639 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:06.322600 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z"] Nov 03 19:45:06.433074 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:06.433041 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d"] Nov 03 19:45:11.323143 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.323079 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" podUID="cdd8a866-c4df-4295-b700-fd2d7a1de211" containerName="payload-processing" containerID="cri-o://31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87" gracePeriod=25 Nov 03 19:45:11.434191 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.434149 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" podUID="b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" containerName="payload-pre-processing" containerID="cri-o://296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81" gracePeriod=25 Nov 03 19:45:11.595225 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.595198 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:45:11.671408 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.671374 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd8a866-c4df-4295-b700-fd2d7a1de211-kube-api-access-74t95\" (UniqueName: \"kubernetes.io/projected/cdd8a866-c4df-4295-b700-fd2d7a1de211-kube-api-access-74t95\") pod \"cdd8a866-c4df-4295-b700-fd2d7a1de211\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " Nov 03 19:45:11.671612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.671513 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd8a866-c4df-4295-b700-fd2d7a1de211-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8a866-c4df-4295-b700-fd2d7a1de211-plugins-config-volume\") pod \"cdd8a866-c4df-4295-b700-fd2d7a1de211\" (UID: \"cdd8a866-c4df-4295-b700-fd2d7a1de211\") " Nov 03 19:45:11.671957 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.671924 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd8a866-c4df-4295-b700-fd2d7a1de211-plugins-config-volume" pod "cdd8a866-c4df-4295-b700-fd2d7a1de211" (UID: "cdd8a866-c4df-4295-b700-fd2d7a1de211"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:11.673522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.673497 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd8a866-c4df-4295-b700-fd2d7a1de211-kube-api-access-74t95" pod "cdd8a866-c4df-4295-b700-fd2d7a1de211" (UID: "cdd8a866-c4df-4295-b700-fd2d7a1de211"). InnerVolumeSpecName "kube-api-access-74t95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:11.689223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.689200 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:45:11.772341 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.772303 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-plugins-config-volume\") pod \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " Nov 03 19:45:11.772557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.772350 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-kube-api-access-xz9vf\" (UniqueName: \"kubernetes.io/projected/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-kube-api-access-xz9vf\") pod \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\" (UID: \"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551\") " Nov 03 19:45:11.772731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.772679 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-plugins-config-volume" pod "b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" (UID: "b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:11.772859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.772692 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74t95\" (UniqueName: \"kubernetes.io/projected/cdd8a866-c4df-4295-b700-fd2d7a1de211-kube-api-access-74t95\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:11.772859 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.772753 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8a866-c4df-4295-b700-fd2d7a1de211-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:11.774522 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.774501 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-kube-api-access-xz9vf" pod "b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" (UID: "b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551"). InnerVolumeSpecName "kube-api-access-xz9vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:11.873805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.873723 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:11.873805 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:11.873755 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9vf\" (UniqueName: \"kubernetes.io/projected/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551-kube-api-access-xz9vf\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:12.319388 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.319352 4948 generic.go:356] "Generic (PLEG): container finished" podID="cdd8a866-c4df-4295-b700-fd2d7a1de211" containerID="31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87" exitCode=0 Nov 03 19:45:12.319599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.319392 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" Nov 03 19:45:12.319599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.319443 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" event={"ID":"cdd8a866-c4df-4295-b700-fd2d7a1de211","Type":"ContainerDied","Data":"31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87"} Nov 03 19:45:12.319599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.319502 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z" event={"ID":"cdd8a866-c4df-4295-b700-fd2d7a1de211","Type":"ContainerDied","Data":"da4e43f14c80ea229f8d64ff27169d0efc6c9261f7277493f3091f1ef02d0a01"} Nov 03 19:45:12.319599 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.319533 4948 scope.go:122] "RemoveContainer" containerID="31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87" Nov 03 19:45:12.321083 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.321050 4948 generic.go:356] "Generic (PLEG): container finished" podID="b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" containerID="296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81" exitCode=0 Nov 03 19:45:12.321219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.321095 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" Nov 03 19:45:12.321219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.321135 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" event={"ID":"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551","Type":"ContainerDied","Data":"296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81"} Nov 03 19:45:12.321219 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.321184 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d" event={"ID":"b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551","Type":"ContainerDied","Data":"fbb57b8d594a90c0999a80109c9381860362d5a85f43da1f190bcccc39a50aba"} Nov 03 19:45:12.330551 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.330309 4948 scope.go:122] "RemoveContainer" containerID="31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87" Nov 03 19:45:12.330920 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:12.330574 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87\": container with ID starting with 31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87 not found: ID does not exist" containerID="31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87" Nov 03 19:45:12.330920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.330601 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87"} err="failed to get container status \"31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87\": rpc error: code = NotFound desc = could not find container \"31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87\": container with ID starting with 31dc4a639b0ebbf28476df045d59cf93153562a3aa99c7f428b368d07e8b8d87 not found: ID does not exist" Nov 03 19:45:12.330920 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.330620 4948 scope.go:122] "RemoveContainer" containerID="296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81" Nov 03 19:45:12.347320 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.347301 4948 scope.go:122] "RemoveContainer" containerID="296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81" Nov 03 19:45:12.347587 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:12.347568 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81\": container with ID starting with 296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81 not found: ID does not exist" containerID="296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81" Nov 03 19:45:12.347650 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.347597 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81"} err="failed to get container status \"296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81\": rpc error: code = NotFound desc = could not find container \"296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81\": container with ID starting with 296e3db033a4002745f0e9e72a614ac52bad9296b77a990f774372653d013a81 not found: ID does not exist" Nov 03 19:45:12.490047 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.490007 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d"] Nov 03 19:45:12.498790 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.498752 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50babf0d-7c8c48b5f5-hg85d"] Nov 03 19:45:12.604803 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.604709 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z"] Nov 03 19:45:12.610155 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.610127 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50babf0d-6db96d59f5-8826z"] Nov 03 19:45:12.752689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.752657 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" path="/var/lib/kubelet/pods/b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551/volumes" Nov 03 19:45:12.753090 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:12.753076 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd8a866-c4df-4295-b700-fd2d7a1de211" path="/var/lib/kubelet/pods/cdd8a866-c4df-4295-b700-fd2d7a1de211/volumes" Nov 03 19:45:14.649900 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:14.649867 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf"] Nov 03 19:45:14.650328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:14.650065 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" podUID="023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" containerName="maas-api" containerID="cri-o://18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc" gracePeriod=30 Nov 03 19:45:14.922658 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:14.922630 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:45:15.001164 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.001131 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-metrics-certs\") pod \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " Nov 03 19:45:15.001368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.001194 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls\") pod \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " Nov 03 19:45:15.001368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.001259 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-kube-api-access-cnv25\" (UniqueName: \"kubernetes.io/projected/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-kube-api-access-cnv25\") pod \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " Nov 03 19:45:15.001368 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.001316 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-openshift-service-ca\") pod \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\" (UID: \"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8\") " Nov 03 19:45:15.001764 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.001737 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-openshift-service-ca" pod "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" (UID: "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:15.003264 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.003239 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-metrics-certs" pod "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" (UID: "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:45:15.003384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.003315 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-kube-api-access-cnv25" pod "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" (UID: "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8"). InnerVolumeSpecName "kube-api-access-cnv25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:15.003384 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.003333 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls" pod "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" (UID: "023059f7-e50d-4f01-b7ee-3a1d87c9f0a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:45:15.102757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.102724 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:15.102757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.102753 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnv25\" (UniqueName: \"kubernetes.io/projected/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-kube-api-access-cnv25\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:15.102757 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.102763 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:15.103004 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.102772 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:15.344099 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.344062 4948 generic.go:356] "Generic (PLEG): container finished" podID="023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" containerID="18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc" exitCode=0 Nov 03 19:45:15.344272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.344113 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" Nov 03 19:45:15.344272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.344147 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" event={"ID":"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8","Type":"ContainerDied","Data":"18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc"} Nov 03 19:45:15.344272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.344200 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf" event={"ID":"023059f7-e50d-4f01-b7ee-3a1d87c9f0a8","Type":"ContainerDied","Data":"a26a25954fe4b36cd18fb8c7184cee730a03dc25a1a4cdc5e1bbc2423d041fcd"} Nov 03 19:45:15.344272 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.344228 4948 scope.go:122] "RemoveContainer" containerID="18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc" Nov 03 19:45:15.355552 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.355535 4948 scope.go:122] "RemoveContainer" containerID="18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc" Nov 03 19:45:15.355837 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:15.355819 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc\": container with ID starting with 18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc not found: ID does not exist" containerID="18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc" Nov 03 19:45:15.355899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.355846 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc"} err="failed to get container status \"18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc\": rpc error: code = NotFound desc = could not find container \"18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc\": container with ID starting with 18f70633d597411166bb42dc22eef886cc6ef1ebc7e60d72c520f41cba14badc not found: ID does not exist" Nov 03 19:45:15.511778 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.511742 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf"] Nov 03 19:45:15.562520 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:15.562486 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50babf0d-5bfb57d569-t2ctf"] Nov 03 19:45:16.753062 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:16.753022 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" path="/var/lib/kubelet/pods/023059f7-e50d-4f01-b7ee-3a1d87c9f0a8/volumes" Nov 03 19:45:21.008243 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:21.008206 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf"] Nov 03 19:45:21.008816 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:21.008456 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" containerID="cri-o://dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df" gracePeriod=30 Nov 03 19:45:22.772887 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.772854 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt"] Nov 03 19:45:22.773338 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773324 4948 cpu_manager.go:414] "containerMap: removing container" podUID="b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" containerName="payload-pre-processing" Nov 03 19:45:22.773385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773339 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" containerName="payload-pre-processing" Nov 03 19:45:22.773385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773351 4948 cpu_manager.go:414] "containerMap: removing container" podUID="023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" containerName="maas-api" Nov 03 19:45:22.773385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773356 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" containerName="maas-api" Nov 03 19:45:22.773385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773370 4948 cpu_manager.go:414] "containerMap: removing container" podUID="cdd8a866-c4df-4295-b700-fd2d7a1de211" containerName="payload-processing" Nov 03 19:45:22.773385 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773375 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd8a866-c4df-4295-b700-fd2d7a1de211" containerName="payload-processing" Nov 03 19:45:22.773550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773446 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bf6be9-2b0c-4b58-8bd1-d9e1d7e66551" containerName="payload-pre-processing" Nov 03 19:45:22.773550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773455 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="023059f7-e50d-4f01-b7ee-3a1d87c9f0a8" containerName="maas-api" Nov 03 19:45:22.773550 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.773464 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd8a866-c4df-4295-b700-fd2d7a1de211" containerName="payload-processing" Nov 03 19:45:22.777687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.777666 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.852950 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.852906 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-ad43e7-openshift-default-dockercfg-n2gmw\"" Nov 03 19:45:22.883221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883182 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-token\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883226 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cb7268-16ad-4e71-8196-4164ac0f4fad-istiod-ca-cert\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883400 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbgd6\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-kube-api-access-rbgd6\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883447 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-socket\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883525 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-podinfo\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883559 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-envoy\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883667 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-data\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883738 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-credential-socket\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.883898 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.883768 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-certs\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.906577 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.906544 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt"] Nov 03 19:45:22.984689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984650 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbgd6\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-kube-api-access-rbgd6\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.984689 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984688 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-socket\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.984969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984751 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-podinfo\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.984969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984789 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-envoy\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.984969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984839 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-data\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.984969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984872 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-credential-socket\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.984969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984893 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-certs\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.984969 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984959 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-token\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.985345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.984987 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cb7268-16ad-4e71-8196-4164ac0f4fad-istiod-ca-cert\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.985345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.985229 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-socket\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.985345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.985256 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-data\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.985345 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.985307 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-certs\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.985591 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.985563 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-credential-socket\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.985826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.985802 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cb7268-16ad-4e71-8196-4164ac0f4fad-istiod-ca-cert\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.987242 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.987221 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-podinfo\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:22.987509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:22.987489 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-envoy\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:23.046328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:23.046266 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-token\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:23.046452 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:23.046380 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbgd6\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-kube-api-access-rbgd6\") pod \"e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:23.091199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:23.091170 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:23.525303 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:23.525274 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt"] Nov 03 19:45:23.526350 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:45:23.526320 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cb7268_16ad_4e71_8196_4164ac0f4fad.slice/crio-a2971eaa84ee76748b63e8e567cf8a17c07c2f14a3c56affba6e13f0e503d127 WatchSource:0}: Error finding container a2971eaa84ee76748b63e8e567cf8a17c07c2f14a3c56affba6e13f0e503d127: Status 404 returned error can't find the container with id a2971eaa84ee76748b63e8e567cf8a17c07c2f14a3c56affba6e13f0e503d127 Nov 03 19:45:23.528549 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:23.528519 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:23.528620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:23.528580 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:23.528620 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:23.528610 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:24.409724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:24.409661 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" event={"ID":"35cb7268-16ad-4e71-8196-4164ac0f4fad","Type":"ContainerStarted","Data":"dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db"} Nov 03 19:45:24.410123 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:24.409784 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" event={"ID":"35cb7268-16ad-4e71-8196-4164ac0f4fad","Type":"ContainerStarted","Data":"a2971eaa84ee76748b63e8e567cf8a17c07c2f14a3c56affba6e13f0e503d127"} Nov 03 19:45:24.535019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:24.534964 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" podStartSLOduration=2.5349497039999997 podStartE2EDuration="2.534949704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:24.532120893 +0000 UTC m=+4054.015116806" watchObservedRunningTime="2026-11-03 19:45:24.534949704 +0000 UTC m=+4054.017945644" Nov 03 19:45:25.092149 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:25.092111 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:25.097423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:25.097396 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:25.416093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:25.416022 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:25.417161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:25.417139 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:26.261182 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.261154 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:45:26.319159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319081 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-socket\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319113 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-credential-socket\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319157 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istiod-ca-cert\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319181 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-envoy\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319206 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-data\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319233 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-kube-api-access-vjpq6\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-kube-api-access-vjpq6\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319297 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-certs\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319339 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-token\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-token\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319389 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-socket" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:45:26.319828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319488 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-credential-socket" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:45:26.319828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319532 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-podinfo\") pod \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\" (UID: \"a506cae3-49d9-4b6d-a9da-e8ef55369c7f\") " Nov 03 19:45:26.319828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319555 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-data" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:45:26.319828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319618 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istiod-ca-cert" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:26.319828 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319671 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-certs" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:45:26.320101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319968 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.320101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319985 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.320101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.319995 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.320101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.320003 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.320101 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.320011 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.321543 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.321513 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-envoy" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:45:26.321672 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.321609 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-token" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:26.321779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.321692 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-podinfo" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:45:26.321779 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.321751 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-kube-api-access-vjpq6" pod "a506cae3-49d9-4b6d-a9da-e8ef55369c7f" (UID: "a506cae3-49d9-4b6d-a9da-e8ef55369c7f"). InnerVolumeSpecName "kube-api-access-vjpq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:26.421010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.420977 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.421010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.421004 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.421010 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.421014 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.421458 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.421025 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjpq6\" (UniqueName: \"kubernetes.io/projected/a506cae3-49d9-4b6d-a9da-e8ef55369c7f-kube-api-access-vjpq6\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:26.423908 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.423878 4948 generic.go:356] "Generic (PLEG): container finished" podID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerID="dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df" exitCode=0 Nov 03 19:45:26.424049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.423928 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" Nov 03 19:45:26.424049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.423962 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" event={"ID":"a506cae3-49d9-4b6d-a9da-e8ef55369c7f","Type":"ContainerDied","Data":"dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df"} Nov 03 19:45:26.424049 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.424031 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf" event={"ID":"a506cae3-49d9-4b6d-a9da-e8ef55369c7f","Type":"ContainerDied","Data":"b65abc5fddc91c30a315419177baab5ad02c96b3470aa802f61ac823b3242bc0"} Nov 03 19:45:26.424190 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.424056 4948 scope.go:122] "RemoveContainer" containerID="dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df" Nov 03 19:45:26.435407 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.435382 4948 scope.go:122] "RemoveContainer" containerID="e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91" Nov 03 19:45:26.446891 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.446870 4948 scope.go:122] "RemoveContainer" containerID="dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df" Nov 03 19:45:26.447171 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:26.447150 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df\": container with ID starting with dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df not found: ID does not exist" containerID="dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df" Nov 03 19:45:26.447276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.447179 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df"} err="failed to get container status \"dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df\": rpc error: code = NotFound desc = could not find container \"dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df\": container with ID starting with dab25012234603a9ef47a83e280d27ea578c087ff4f7d177f9fb41d636fd66df not found: ID does not exist" Nov 03 19:45:26.447276 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.447208 4948 scope.go:122] "RemoveContainer" containerID="e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91" Nov 03 19:45:26.447489 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:26.447471 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91\": container with ID starting with e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91 not found: ID does not exist" containerID="e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91" Nov 03 19:45:26.447560 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.447493 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91"} err="failed to get container status \"e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91\": rpc error: code = NotFound desc = could not find container \"e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91\": container with ID starting with e32f13375e9645c3d6b5c02644729b4e1e408438a15962d623f4788175ee5d91 not found: ID does not exist" Nov 03 19:45:26.549750 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.549719 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf"] Nov 03 19:45:26.558972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.558937 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-50babf0d-openshift-default-6bc768cbf7-7j7jf"] Nov 03 19:45:26.752328 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:26.752295 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" path="/var/lib/kubelet/pods/a506cae3-49d9-4b6d-a9da-e8ef55369c7f/volumes" Nov 03 19:45:31.035468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.035433 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt"] Nov 03 19:45:31.035972 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.035952 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" Nov 03 19:45:31.036043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.035973 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" Nov 03 19:45:31.036043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.036013 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" Nov 03 19:45:31.036043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.036020 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" Nov 03 19:45:31.036156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.036093 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" Nov 03 19:45:31.039501 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.039482 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.068886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.068859 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6x55\" (UniqueName: \"kubernetes.io/projected/874aa671-864c-4b9b-a69c-e1b473aa08c2-kube-api-access-n6x55\") pod \"payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.069033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.068923 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874aa671-864c-4b9b-a69c-e1b473aa08c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.093775 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.093742 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt"] Nov 03 19:45:31.141886 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.141848 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-ad43e7-dockercfg-k5jng\"" Nov 03 19:45:31.142415 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.142351 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-ad43e7\"" Nov 03 19:45:31.169845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.169806 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6x55\" (UniqueName: \"kubernetes.io/projected/874aa671-864c-4b9b-a69c-e1b473aa08c2-kube-api-access-n6x55\") pod \"payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.170040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.169873 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874aa671-864c-4b9b-a69c-e1b473aa08c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.170346 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.170324 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd"] Nov 03 19:45:31.170621 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.170600 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874aa671-864c-4b9b-a69c-e1b473aa08c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.170921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.170904 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a506cae3-49d9-4b6d-a9da-e8ef55369c7f" containerName="istio-proxy" Nov 03 19:45:31.174257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.174235 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.228852 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.228821 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd"] Nov 03 19:45:31.233641 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.233608 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6x55\" (UniqueName: \"kubernetes.io/projected/874aa671-864c-4b9b-a69c-e1b473aa08c2-kube-api-access-n6x55\") pod \"payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.271405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.271375 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1216613-b5bf-4235-afb3-fdaf0d072f1c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.271579 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.271434 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf67c\" (UniqueName: \"kubernetes.io/projected/a1216613-b5bf-4235-afb3-fdaf0d072f1c-kube-api-access-zf67c\") pod \"payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.353100 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.353023 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:31.372561 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.372520 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1216613-b5bf-4235-afb3-fdaf0d072f1c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.372680 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.372609 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf67c\" (UniqueName: \"kubernetes.io/projected/a1216613-b5bf-4235-afb3-fdaf0d072f1c-kube-api-access-zf67c\") pod \"payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.373448 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.373425 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1216613-b5bf-4235-afb3-fdaf0d072f1c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.434826 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.434794 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf67c\" (UniqueName: \"kubernetes.io/projected/a1216613-b5bf-4235-afb3-fdaf0d072f1c-kube-api-access-zf67c\") pod \"payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.488509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.488475 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:31.730798 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.730587 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt"] Nov 03 19:45:31.733624 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:45:31.733596 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874aa671_864c_4b9b_a69c_e1b473aa08c2.slice/crio-d2a572538b818c76573195dab81da77ad97d111aa922737eb36283950562354d WatchSource:0}: Error finding container d2a572538b818c76573195dab81da77ad97d111aa922737eb36283950562354d: Status 404 returned error can't find the container with id d2a572538b818c76573195dab81da77ad97d111aa922737eb36283950562354d Nov 03 19:45:31.870076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:31.870043 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd"] Nov 03 19:45:31.871549 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:45:31.871525 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1216613_b5bf_4235_afb3_fdaf0d072f1c.slice/crio-e9ebe6cc767e6d417bd37422d77657924a59eac721770452eeadc22acf5bc9ce WatchSource:0}: Error finding container e9ebe6cc767e6d417bd37422d77657924a59eac721770452eeadc22acf5bc9ce: Status 404 returned error can't find the container with id e9ebe6cc767e6d417bd37422d77657924a59eac721770452eeadc22acf5bc9ce Nov 03 19:45:32.463053 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.463010 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" event={"ID":"a1216613-b5bf-4235-afb3-fdaf0d072f1c","Type":"ContainerStarted","Data":"58614b6e89d87ab38d7caea8efa40c0321843e81ca412b5db2a31452da0422bd"} Nov 03 19:45:32.463533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.463072 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" event={"ID":"a1216613-b5bf-4235-afb3-fdaf0d072f1c","Type":"ContainerStarted","Data":"e9ebe6cc767e6d417bd37422d77657924a59eac721770452eeadc22acf5bc9ce"} Nov 03 19:45:32.463533 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.463124 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:32.464771 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.464745 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" event={"ID":"874aa671-864c-4b9b-a69c-e1b473aa08c2","Type":"ContainerStarted","Data":"5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400"} Nov 03 19:45:32.464892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.464787 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" event={"ID":"874aa671-864c-4b9b-a69c-e1b473aa08c2","Type":"ContainerStarted","Data":"d2a572538b818c76573195dab81da77ad97d111aa922737eb36283950562354d"} Nov 03 19:45:32.464892 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.464814 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:32.581940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.581886 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" podStartSLOduration=1.581873577 podStartE2EDuration="1.581873577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:32.580333566 +0000 UTC m=+4062.063329480" watchObservedRunningTime="2026-11-03 19:45:32.581873577 +0000 UTC m=+4062.064869491" Nov 03 19:45:32.697382 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:32.697318 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" podStartSLOduration=2.697298653 podStartE2EDuration="2.697298653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:32.696925008 +0000 UTC m=+4062.179920924" watchObservedRunningTime="2026-11-03 19:45:32.697298653 +0000 UTC m=+4062.180294568" Nov 03 19:45:32.793321 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:32.793239 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:45:33.649582 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.649546 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p"] Nov 03 19:45:33.653509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.653490 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.696125 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.696099 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2739f09d-06b4-40c2-a7d6-28a7f511feeb-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.696260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.696135 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptv7\" (UniqueName: \"kubernetes.io/projected/2739f09d-06b4-40c2-a7d6-28a7f511feeb-kube-api-access-5ptv7\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.696260 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.696245 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.696348 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.696275 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.706632 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.706604 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-ad43e7-serving-cert\"" Nov 03 19:45:33.709491 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.709465 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p"] Nov 03 19:45:33.797534 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.797499 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.797756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.797549 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.797756 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.797660 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2739f09d-06b4-40c2-a7d6-28a7f511feeb-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.797756 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:33.797682 4948 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-serving-cert: secret "maas-api-e2e-authcfg-http-ad43e7-serving-cert" not found Nov 03 19:45:33.797958 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:45:33.797774 4948 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls podName:2739f09d-06b4-40c2-a7d6-28a7f511feeb nodeName:}" failed. No retries permitted until 2026-11-03 19:45:34.297753868 +0000 UTC m=+4063.780749776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls") pod "maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" (UID: "2739f09d-06b4-40c2-a7d6-28a7f511feeb") : secret "maas-api-e2e-authcfg-http-ad43e7-serving-cert" not found Nov 03 19:45:33.797958 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.797687 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptv7\" (UniqueName: \"kubernetes.io/projected/2739f09d-06b4-40c2-a7d6-28a7f511feeb-kube-api-access-5ptv7\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.798544 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.798521 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2739f09d-06b4-40c2-a7d6-28a7f511feeb-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.800212 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.800182 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:33.861855 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:33.861826 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptv7\" (UniqueName: \"kubernetes.io/projected/2739f09d-06b4-40c2-a7d6-28a7f511feeb-kube-api-access-5ptv7\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:34.304832 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:34.304796 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:34.307258 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:34.307237 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:34.569022 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:34.568931 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:34.951037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:34.950972 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p"] Nov 03 19:45:34.951657 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:45:34.951619 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2739f09d_06b4_40c2_a7d6_28a7f511feeb.slice/crio-266b42597977934c8bcde8e5a8105118ee01bd9983611e09bc3b195b05c9e3da WatchSource:0}: Error finding container 266b42597977934c8bcde8e5a8105118ee01bd9983611e09bc3b195b05c9e3da: Status 404 returned error can't find the container with id 266b42597977934c8bcde8e5a8105118ee01bd9983611e09bc3b195b05c9e3da Nov 03 19:45:35.485033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:35.484992 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" event={"ID":"2739f09d-06b4-40c2-a7d6-28a7f511feeb","Type":"ContainerStarted","Data":"266b42597977934c8bcde8e5a8105118ee01bd9983611e09bc3b195b05c9e3da"} Nov 03 19:45:36.492662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:36.492620 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" event={"ID":"2739f09d-06b4-40c2-a7d6-28a7f511feeb","Type":"ContainerStarted","Data":"c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e"} Nov 03 19:45:36.493189 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:36.492691 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:36.613597 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:36.613540 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" podStartSLOduration=2.755585515 podStartE2EDuration="3.613525378s" totalImagesPullingTime="857.939863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:36.611661864 +0000 UTC m=+4066.094657778" watchObservedRunningTime="2026-11-03 19:45:36.613525378 +0000 UTC m=+4066.096521291" Nov 03 19:45:42.505416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:42.505386 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:45:43.479524 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:43.479486 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:45:43.479768 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:43.479569 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:45:51.611426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:51.611388 4948 generic.go:356] "Generic (PLEG): container finished" podID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerID="dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db" exitCode=0 Nov 03 19:45:51.611932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:51.611463 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" event={"ID":"35cb7268-16ad-4e71-8196-4164ac0f4fad","Type":"ContainerDied","Data":"dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db"} Nov 03 19:45:51.611932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:51.611899 4948 scope.go:122] "RemoveContainer" containerID="dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db" Nov 03 19:45:51.613226 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:51.613191 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:51.613330 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:51.613285 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:51.613374 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:51.613330 4948 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:52.619869 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:52.619831 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" event={"ID":"35cb7268-16ad-4e71-8196-4164ac0f4fad","Type":"ContainerStarted","Data":"c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81"} Nov 03 19:45:53.092176 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:53.092131 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:53.093667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:53.093633 4948 patch_prober.go:29] interesting pod/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.093824 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:53.093735 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 19:45:54.091576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:54.091542 4948 patch_prober.go:29] interesting pod/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 19:45:54.092054 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:54.091593 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 19:45:55.092557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:55.092521 4948 patch_prober.go:29] interesting pod/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 19:45:55.092978 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:55.092578 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 19:45:55.416897 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:55.416810 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:56.092033 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:56.091998 4948 patch_prober.go:29] interesting pod/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 19:45:56.092223 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:56.092060 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 19:45:57.091939 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:57.091899 4948 patch_prober.go:29] interesting pod/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 19:45:57.092337 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:57.091959 4948 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 19:45:58.096557 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:58.096525 4948 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:45:58.097652 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:45:58.097628 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:46:01.793509 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:46:01.793437 4948 scope.go:122] "RemoveContainer" containerID="f3903e7627a83f9456b20071040d6ca3649e28f26e42907e85f3be05bc9d38be" Nov 03 19:46:01.803012 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:46:01.802989 4948 scope.go:122] "RemoveContainer" containerID="aa70f77fa2353f59cd0580eef14d0bf64c74da548e8107a738db62b75be50bf8" Nov 03 19:46:02.792187 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:46:02.792151 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:46:32.792990 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:46:32.792954 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:47:02.793080 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:47:02.793042 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:47:24.036065 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:24.036028 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd"] Nov 03 19:47:24.146048 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:24.146014 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt"] Nov 03 19:47:29.037006 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.036912 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" podUID="a1216613-b5bf-4235-afb3-fdaf0d072f1c" containerName="payload-processing" containerID="cri-o://58614b6e89d87ab38d7caea8efa40c0321843e81ca412b5db2a31452da0422bd" gracePeriod=25 Nov 03 19:47:29.146874 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.146836 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" podUID="874aa671-864c-4b9b-a69c-e1b473aa08c2" containerName="payload-pre-processing" containerID="cri-o://5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400" gracePeriod=25 Nov 03 19:47:29.231201 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.231161 4948 generic.go:356] "Generic (PLEG): container finished" podID="a1216613-b5bf-4235-afb3-fdaf0d072f1c" containerID="58614b6e89d87ab38d7caea8efa40c0321843e81ca412b5db2a31452da0422bd" exitCode=0 Nov 03 19:47:29.231390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.231209 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" event={"ID":"a1216613-b5bf-4235-afb3-fdaf0d072f1c","Type":"ContainerDied","Data":"58614b6e89d87ab38d7caea8efa40c0321843e81ca412b5db2a31452da0422bd"} Nov 03 19:47:29.299175 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.299109 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:47:29.405410 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.405387 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:47:29.444983 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.444936 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1216613-b5bf-4235-afb3-fdaf0d072f1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1216613-b5bf-4235-afb3-fdaf0d072f1c-plugins-config-volume\") pod \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " Nov 03 19:47:29.445161 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.445004 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1216613-b5bf-4235-afb3-fdaf0d072f1c-kube-api-access-zf67c\" (UniqueName: \"kubernetes.io/projected/a1216613-b5bf-4235-afb3-fdaf0d072f1c-kube-api-access-zf67c\") pod \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\" (UID: \"a1216613-b5bf-4235-afb3-fdaf0d072f1c\") " Nov 03 19:47:29.445423 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.445397 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1216613-b5bf-4235-afb3-fdaf0d072f1c-plugins-config-volume" pod "a1216613-b5bf-4235-afb3-fdaf0d072f1c" (UID: "a1216613-b5bf-4235-afb3-fdaf0d072f1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:29.447325 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.447294 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1216613-b5bf-4235-afb3-fdaf0d072f1c-kube-api-access-zf67c" pod "a1216613-b5bf-4235-afb3-fdaf0d072f1c" (UID: "a1216613-b5bf-4235-afb3-fdaf0d072f1c"). InnerVolumeSpecName "kube-api-access-zf67c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:29.546792 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.546744 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874aa671-864c-4b9b-a69c-e1b473aa08c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874aa671-864c-4b9b-a69c-e1b473aa08c2-plugins-config-volume\") pod \"874aa671-864c-4b9b-a69c-e1b473aa08c2\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " Nov 03 19:47:29.546947 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.546856 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874aa671-864c-4b9b-a69c-e1b473aa08c2-kube-api-access-n6x55\" (UniqueName: \"kubernetes.io/projected/874aa671-864c-4b9b-a69c-e1b473aa08c2-kube-api-access-n6x55\") pod \"874aa671-864c-4b9b-a69c-e1b473aa08c2\" (UID: \"874aa671-864c-4b9b-a69c-e1b473aa08c2\") " Nov 03 19:47:29.547122 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.547100 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874aa671-864c-4b9b-a69c-e1b473aa08c2-plugins-config-volume" pod "874aa671-864c-4b9b-a69c-e1b473aa08c2" (UID: "874aa671-864c-4b9b-a69c-e1b473aa08c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:29.547227 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.547213 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1216613-b5bf-4235-afb3-fdaf0d072f1c-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:29.547273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.547230 4948 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874aa671-864c-4b9b-a69c-e1b473aa08c2-plugins-config-volume\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:29.547273 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.547239 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf67c\" (UniqueName: \"kubernetes.io/projected/a1216613-b5bf-4235-afb3-fdaf0d072f1c-kube-api-access-zf67c\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:29.548975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.548941 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874aa671-864c-4b9b-a69c-e1b473aa08c2-kube-api-access-n6x55" pod "874aa671-864c-4b9b-a69c-e1b473aa08c2" (UID: "874aa671-864c-4b9b-a69c-e1b473aa08c2"). InnerVolumeSpecName "kube-api-access-n6x55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:29.647998 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:29.647908 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6x55\" (UniqueName: \"kubernetes.io/projected/874aa671-864c-4b9b-a69c-e1b473aa08c2-kube-api-access-n6x55\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:30.238715 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.238666 4948 generic.go:356] "Generic (PLEG): container finished" podID="874aa671-864c-4b9b-a69c-e1b473aa08c2" containerID="5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400" exitCode=0 Nov 03 19:47:30.239179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.238726 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" event={"ID":"874aa671-864c-4b9b-a69c-e1b473aa08c2","Type":"ContainerDied","Data":"5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400"} Nov 03 19:47:30.239179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.238747 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" Nov 03 19:47:30.239179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.238783 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt" event={"ID":"874aa671-864c-4b9b-a69c-e1b473aa08c2","Type":"ContainerDied","Data":"d2a572538b818c76573195dab81da77ad97d111aa922737eb36283950562354d"} Nov 03 19:47:30.239179 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.238805 4948 scope.go:122] "RemoveContainer" containerID="5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400" Nov 03 19:47:30.240439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.240415 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" event={"ID":"a1216613-b5bf-4235-afb3-fdaf0d072f1c","Type":"ContainerDied","Data":"e9ebe6cc767e6d417bd37422d77657924a59eac721770452eeadc22acf5bc9ce"} Nov 03 19:47:30.240568 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.240459 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd" Nov 03 19:47:30.248604 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.248587 4948 scope.go:122] "RemoveContainer" containerID="5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400" Nov 03 19:47:30.248973 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:47:30.248949 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400\": container with ID starting with 5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400 not found: ID does not exist" containerID="5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400" Nov 03 19:47:30.249071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.248985 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400"} err="failed to get container status \"5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400\": rpc error: code = NotFound desc = could not find container \"5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400\": container with ID starting with 5a178331217e0eb00e42cdae9dab228f82fd105c281b496cc45d679bbc0fb400 not found: ID does not exist" Nov 03 19:47:30.249071 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.249009 4948 scope.go:122] "RemoveContainer" containerID="58614b6e89d87ab38d7caea8efa40c0321843e81ca412b5db2a31452da0422bd" Nov 03 19:47:30.407113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.407005 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd"] Nov 03 19:47:30.457804 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.457767 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ad43e7-66b7bc6d87-gn4qd"] Nov 03 19:47:30.521014 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.520981 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt"] Nov 03 19:47:30.523943 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.523923 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ad43e7-7d7956b4ff-xwtpt"] Nov 03 19:47:30.753463 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.753388 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874aa671-864c-4b9b-a69c-e1b473aa08c2" path="/var/lib/kubelet/pods/874aa671-864c-4b9b-a69c-e1b473aa08c2/volumes" Nov 03 19:47:30.753823 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:30.753809 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1216613-b5bf-4235-afb3-fdaf0d072f1c" path="/var/lib/kubelet/pods/a1216613-b5bf-4235-afb3-fdaf0d072f1c/volumes" Nov 03 19:47:32.793569 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:47:32.793531 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:47:34.354948 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.354916 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p"] Nov 03 19:47:34.355454 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.355180 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" podUID="2739f09d-06b4-40c2-a7d6-28a7f511feeb" containerName="maas-api" containerID="cri-o://c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e" gracePeriod=30 Nov 03 19:47:34.622006 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.621950 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:47:34.696882 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.696837 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2739f09d-06b4-40c2-a7d6-28a7f511feeb-kube-api-access-5ptv7\" (UniqueName: \"kubernetes.io/projected/2739f09d-06b4-40c2-a7d6-28a7f511feeb-kube-api-access-5ptv7\") pod \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " Nov 03 19:47:34.697040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.696910 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls\") pod \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " Nov 03 19:47:34.697040 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.697013 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2739f09d-06b4-40c2-a7d6-28a7f511feeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2739f09d-06b4-40c2-a7d6-28a7f511feeb-openshift-service-ca\") pod \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " Nov 03 19:47:34.697162 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.697041 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-metrics-certs\") pod \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\" (UID: \"2739f09d-06b4-40c2-a7d6-28a7f511feeb\") " Nov 03 19:47:34.697387 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.697353 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2739f09d-06b4-40c2-a7d6-28a7f511feeb-openshift-service-ca" pod "2739f09d-06b4-40c2-a7d6-28a7f511feeb" (UID: "2739f09d-06b4-40c2-a7d6-28a7f511feeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:34.697510 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.697476 4948 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2739f09d-06b4-40c2-a7d6-28a7f511feeb-openshift-service-ca\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:34.699037 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.699010 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls" pod "2739f09d-06b4-40c2-a7d6-28a7f511feeb" (UID: "2739f09d-06b4-40c2-a7d6-28a7f511feeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:34.699152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.699084 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-metrics-certs" pod "2739f09d-06b4-40c2-a7d6-28a7f511feeb" (UID: "2739f09d-06b4-40c2-a7d6-28a7f511feeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:34.699152 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.699121 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2739f09d-06b4-40c2-a7d6-28a7f511feeb-kube-api-access-5ptv7" pod "2739f09d-06b4-40c2-a7d6-28a7f511feeb" (UID: "2739f09d-06b4-40c2-a7d6-28a7f511feeb"). InnerVolumeSpecName "kube-api-access-5ptv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:34.798607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.798574 4948 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-metrics-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:34.798607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.798601 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptv7\" (UniqueName: \"kubernetes.io/projected/2739f09d-06b4-40c2-a7d6-28a7f511feeb-kube-api-access-5ptv7\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:34.798607 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:34.798610 4948 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2739f09d-06b4-40c2-a7d6-28a7f511feeb-maas-api-tls\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:35.273987 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.273954 4948 generic.go:356] "Generic (PLEG): container finished" podID="2739f09d-06b4-40c2-a7d6-28a7f511feeb" containerID="c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e" exitCode=0 Nov 03 19:47:35.274204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.274003 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" Nov 03 19:47:35.274204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.274032 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" event={"ID":"2739f09d-06b4-40c2-a7d6-28a7f511feeb","Type":"ContainerDied","Data":"c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e"} Nov 03 19:47:35.274204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.274097 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p" event={"ID":"2739f09d-06b4-40c2-a7d6-28a7f511feeb","Type":"ContainerDied","Data":"266b42597977934c8bcde8e5a8105118ee01bd9983611e09bc3b195b05c9e3da"} Nov 03 19:47:35.274204 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.274129 4948 scope.go:122] "RemoveContainer" containerID="c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e" Nov 03 19:47:35.283627 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.283609 4948 scope.go:122] "RemoveContainer" containerID="c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e" Nov 03 19:47:35.283918 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:47:35.283899 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e\": container with ID starting with c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e not found: ID does not exist" containerID="c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e" Nov 03 19:47:35.283988 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.283925 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e"} err="failed to get container status \"c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e\": rpc error: code = NotFound desc = could not find container \"c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e\": container with ID starting with c9b8c45ec3f54101d48026e783c94d5eac0db98c5563d9a929f08119e3d3114e not found: ID does not exist" Nov 03 19:47:35.443899 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.443865 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p"] Nov 03 19:47:35.448836 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:35.448810 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ad43e7-65d78bccd8-dvk7p"] Nov 03 19:47:36.752742 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:36.752680 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2739f09d-06b4-40c2-a7d6-28a7f511feeb" path="/var/lib/kubelet/pods/2739f09d-06b4-40c2-a7d6-28a7f511feeb/volumes" Nov 03 19:47:43.392662 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:43.392625 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt"] Nov 03 19:47:43.393236 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:43.392936 4948 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" containerID="cri-o://c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81" gracePeriod=30 Nov 03 19:47:48.645253 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.645227 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:47:48.738994 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.738964 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-data\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739028 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-kube-api-access-rbgd6\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-kube-api-access-rbgd6\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739080 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-token\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-token\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739115 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cb7268-16ad-4e71-8196-4164ac0f4fad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cb7268-16ad-4e71-8196-4164ac0f4fad-istiod-ca-cert\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739170 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739158 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-podinfo\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739190 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-envoy\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739224 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-socket\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739268 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-credential-socket\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739264 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-data" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.739403 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739370 4948 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-certs\") pod \"35cb7268-16ad-4e71-8196-4164ac0f4fad\" (UID: \"35cb7268-16ad-4e71-8196-4164ac0f4fad\") " Nov 03 19:47:48.739685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739514 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-credential-socket" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.739685 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739667 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-socket" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.739845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739689 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cb7268-16ad-4e71-8196-4164ac0f4fad-istiod-ca-cert" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:48.739845 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739773 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-certs" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.739941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739855 4948 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-data\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.739941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739876 4948 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cb7268-16ad-4e71-8196-4164ac0f4fad-istiod-ca-cert\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.739941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739891 4948 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.739941 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.739906 4948 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-credential-socket\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.741315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.741284 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-token" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:48.741315 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.741297 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-podinfo" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:47:48.741506 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.741361 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-kube-api-access-rbgd6" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "kube-api-access-rbgd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:48.741681 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.741662 4948 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-envoy" pod "35cb7268-16ad-4e71-8196-4164ac0f4fad" (UID: "35cb7268-16ad-4e71-8196-4164ac0f4fad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.841086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.841048 4948 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-podinfo\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.841086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.841079 4948 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-envoy\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.841086 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.841089 4948 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cb7268-16ad-4e71-8196-4164ac0f4fad-workload-certs\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.841405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.841098 4948 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbgd6\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-kube-api-access-rbgd6\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.841405 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:48.841108 4948 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cb7268-16ad-4e71-8196-4164ac0f4fad-istio-token\") on node \"ip-10-0-128-95.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:49.361142 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.361108 4948 generic.go:356] "Generic (PLEG): container finished" podID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerID="c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81" exitCode=0 Nov 03 19:47:49.361297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.361158 4948 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" Nov 03 19:47:49.361297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.361187 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" event={"ID":"35cb7268-16ad-4e71-8196-4164ac0f4fad","Type":"ContainerDied","Data":"c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81"} Nov 03 19:47:49.361297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.361240 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt" event={"ID":"35cb7268-16ad-4e71-8196-4164ac0f4fad","Type":"ContainerDied","Data":"a2971eaa84ee76748b63e8e567cf8a17c07c2f14a3c56affba6e13f0e503d127"} Nov 03 19:47:49.361297 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.361275 4948 scope.go:122] "RemoveContainer" containerID="c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81" Nov 03 19:47:49.371158 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.371138 4948 scope.go:122] "RemoveContainer" containerID="dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db" Nov 03 19:47:49.379861 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.379841 4948 scope.go:122] "RemoveContainer" containerID="c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81" Nov 03 19:47:49.380103 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:47:49.380084 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81\": container with ID starting with c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81 not found: ID does not exist" containerID="c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81" Nov 03 19:47:49.380160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.380112 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81"} err="failed to get container status \"c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81\": rpc error: code = NotFound desc = could not find container \"c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81\": container with ID starting with c92dbca0d6310a935830ddb6ec8f270f7ea9e88a4c4388552a0f0239b7259e81 not found: ID does not exist" Nov 03 19:47:49.380160 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.380130 4948 scope.go:122] "RemoveContainer" containerID="dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db" Nov 03 19:47:49.380328 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:47:49.380310 4948 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db\": container with ID starting with dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db not found: ID does not exist" containerID="dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db" Nov 03 19:47:49.380366 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.380332 4948 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db"} err="failed to get container status \"dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db\": rpc error: code = NotFound desc = could not find container \"dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db\": container with ID starting with dbaed22d679ada3c986498b8ffc6b069c2a9005c85e06799e5be95dd745dc7db not found: ID does not exist" Nov 03 19:47:49.580307 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.580271 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt"] Nov 03 19:47:49.631390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:49.631315 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ad43e7-openshift-default-5d97569dcc-p2ckt"] Nov 03 19:47:50.753119 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:50.753087 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" path="/var/lib/kubelet/pods/35cb7268-16ad-4e71-8196-4164ac0f4fad/volumes" Nov 03 19:47:55.931992 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:55.931960 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:47:55.941670 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:55.941634 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:47:56.158188 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:56.158163 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5gvn_8f64f0eb-fed7-496b-92a6-84601b0bb544/console-operator/3.log" Nov 03 19:47:56.164687 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:47:56.164667 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t92q5_ce2ce282-8a19-40f3-a97f-a270daedbe66/ovn-acl-logging/0.log" Nov 03 19:48:02.792842 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:48:02.792803 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:48:09.067791 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:09.067088 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr"] Nov 03 19:48:09.109467 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:09.109428 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kktlr"] Nov 03 19:48:10.752334 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:10.752298 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19253e67-a331-4aa8-a63e-90fd04d92567" path="/var/lib/kubelet/pods/19253e67-a331-4aa8-a63e-90fd04d92567/volumes" Nov 03 19:48:29.061447 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:29.061408 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4"] Nov 03 19:48:29.114277 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:29.114239 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gzjp4"] Nov 03 19:48:30.753284 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:30.753245 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb9cbda0-2906-4f82-9a4c-a0ccc1395652" path="/var/lib/kubelet/pods/eb9cbda0-2906-4f82-9a4c-a0ccc1395652/volumes" Nov 03 19:48:32.987303 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:48:32.987261 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:48:58.050910 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:58.050874 4948 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm"] Nov 03 19:48:58.102113 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:58.102079 4948 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d261134f-pdqgm"] Nov 03 19:48:58.752300 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:48:58.752263 4948 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25498b48-7144-4193-8ddb-2cb0959f330c" path="/var/lib/kubelet/pods/25498b48-7144-4193-8ddb-2cb0959f330c/volumes" Nov 03 19:49:02.793864 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:49:02.793768 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:49:09.866240 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:09.866199 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5f84cbd65-zgqhf_444955b0-fba7-45b4-b7ab-bb94095f64df/manager/0.log" Nov 03 19:49:10.504455 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:10.504418 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9rqtr_82be7705-1844-40b4-9349-49951cf7962e/manager/0.log" Nov 03 19:49:11.257667 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:11.257630 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68f76587d6-t8kgt_5feb0c16-6030-4cda-8e9f-243c7a77d023/manager/0.log" Nov 03 19:49:13.326426 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:13.326394 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6696c7b56d-kpmhf_87a80863-a264-4d18-8507-ddc2926a7261/maas-api/0.log" Nov 03 19:49:16.550763 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:16.550728 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pj4wz_7995752d-b5e3-4b41-b7e7-e9893ca31b33/manager/0.log" Nov 03 19:49:16.860428 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:16.860351 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-rfj7j_3bb8de85-fed2-48d4-aeda-c1e45945aeeb/manager/0.log" Nov 03 19:49:17.339916 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:17.339882 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2q8j7_d991fd98-c238-4214-b938-daa90d01841f/manager/0.log" Nov 03 19:49:18.025626 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:18.025595 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nmztb_be9f4c2f-1332-4a3f-a756-96959733fa78/manager/0.log" Nov 03 19:49:18.811921 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:18.811891 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp_38f23fde-7287-45c5-9e3a-31a0ae356e1d/istio-proxy/0.log" Nov 03 19:49:19.881481 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:19.881450 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-8pc6b_4f2486e1-e5ca-46ac-aea5-9101cc6eb10f/istio-proxy/0.log" Nov 03 19:49:20.426168 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:20.426137 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-fmq4c_ab5a14d7-8474-45ff-ae36-87c0aabad985/istio-proxy/0.log" Nov 03 19:49:21.380538 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:21.380466 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-tg88d_cdd296b3-a365-49a1-8fe2-c94e943a7618/istio-proxy/0.log" Nov 03 19:49:21.943783 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:21.943751 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-vdlrm_99a8205c-e0cd-4eeb-828a-c78ec53fe35b/istio-proxy/0.log" Nov 03 19:49:22.509645 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:22.509618 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r_9c2e9bae-7530-4694-aa1a-4fcbfae723c4/istio-proxy/0.log" Nov 03 19:49:22.725932 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:22.725899 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zrpdh_1252e5c7-6c27-4493-8cc3-f1070ba4d5a1/payload-pre-processing/0.log" Nov 03 19:49:22.951011 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:22.950982 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kt9k6_53a4c748-6a4e-46eb-8080-82f904149747/payload-processing/0.log" Nov 03 19:49:23.165157 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:23.165129 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7577979dbf-qm2k5_0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/router/0.log" Nov 03 19:49:24.548711 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:24.548666 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ktxrg_c659d570-af36-4aca-9532-5dc9e24ef622/main/0.log" Nov 03 19:49:25.034679 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:25.034650 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf_621ec304-b99e-4a18-b966-dd7e1e0c73b3/main/0.log" Nov 03 19:49:32.793586 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:49:32.793539 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:49:35.871690 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:35.871660 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5f84cbd65-zgqhf_444955b0-fba7-45b4-b7ab-bb94095f64df/manager/0.log" Nov 03 19:49:36.496406 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:36.496375 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9rqtr_82be7705-1844-40b4-9349-49951cf7962e/manager/0.log" Nov 03 19:49:37.174017 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:37.173982 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68f76587d6-t8kgt_5feb0c16-6030-4cda-8e9f-243c7a77d023/manager/0.log" Nov 03 19:49:38.578076 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:38.578040 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6696c7b56d-kpmhf_87a80863-a264-4d18-8507-ddc2926a7261/maas-api/0.log" Nov 03 19:49:41.568928 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:41.568896 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pj4wz_7995752d-b5e3-4b41-b7e7-e9893ca31b33/manager/0.log" Nov 03 19:49:41.878990 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:41.878898 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-rfj7j_3bb8de85-fed2-48d4-aeda-c1e45945aeeb/manager/0.log" Nov 03 19:49:42.336359 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:42.336311 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2q8j7_d991fd98-c238-4214-b938-daa90d01841f/manager/0.log" Nov 03 19:49:42.923361 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:42.923329 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nmztb_be9f4c2f-1332-4a3f-a756-96959733fa78/manager/0.log" Nov 03 19:49:43.520508 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:43.520477 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp_38f23fde-7287-45c5-9e3a-31a0ae356e1d/istio-proxy/0.log" Nov 03 19:49:44.551967 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:44.551939 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-8pc6b_4f2486e1-e5ca-46ac-aea5-9101cc6eb10f/istio-proxy/0.log" Nov 03 19:49:45.121488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:45.121407 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-fmq4c_ab5a14d7-8474-45ff-ae36-87c0aabad985/istio-proxy/0.log" Nov 03 19:49:46.129974 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:46.129941 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-tg88d_cdd296b3-a365-49a1-8fe2-c94e943a7618/istio-proxy/0.log" Nov 03 19:49:46.676935 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:46.676903 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-vdlrm_99a8205c-e0cd-4eeb-828a-c78ec53fe35b/istio-proxy/0.log" Nov 03 19:49:47.214286 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:47.214255 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-qbp4r_9c2e9bae-7530-4694-aa1a-4fcbfae723c4/istio-proxy/0.log" Nov 03 19:49:47.439875 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:47.439847 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zrpdh_1252e5c7-6c27-4493-8cc3-f1070ba4d5a1/payload-pre-processing/0.log" Nov 03 19:49:47.665468 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:47.665391 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kt9k6_53a4c748-6a4e-46eb-8080-82f904149747/payload-processing/0.log" Nov 03 19:49:47.886547 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:47.886505 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7577979dbf-qm2k5_0d1827e8-4e13-48b4-a80f-4b2ef3d3d2e9/router/0.log" Nov 03 19:49:49.056762 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:49.056728 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ktxrg_c659d570-af36-4aca-9532-5dc9e24ef622/main/0.log" Nov 03 19:49:49.549094 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:49:49.549061 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtfkf_621ec304-b99e-4a18-b966-dd7e1e0c73b3/main/0.log" Nov 03 19:50:01.920540 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:01.920501 4948 scope.go:122] "RemoveContainer" containerID="46f40ed5aa388694729aff57167165c383f83cd57ce20d8d5f3e6488319a3676" Nov 03 19:50:01.929975 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:01.929952 4948 scope.go:122] "RemoveContainer" containerID="9489082e7fd7340a5b9a1cc956e60b9b7e18a586adfe34c9819bbaad3ec25a68" Nov 03 19:50:01.939002 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:01.938975 4948 scope.go:122] "RemoveContainer" containerID="2a2660c4c431efd39070e1a9e2417e9f0ffeb515c1be80396e013a51f9310d4d" Nov 03 19:50:02.793172 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:50:02.793131 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:50:17.995500 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:17.995459 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68f76587d6-t8kgt_5feb0c16-6030-4cda-8e9f-243c7a77d023/manager/0.log" Nov 03 19:50:18.538717 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:18.538673 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68f76587d6-t8kgt_5feb0c16-6030-4cda-8e9f-243c7a77d023/manager/0.log" Nov 03 19:50:19.303761 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:19.303729 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kt9k6_53a4c748-6a4e-46eb-8080-82f904149747/payload-processing/0.log" Nov 03 19:50:19.517646 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:19.517613 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kt9k6_53a4c748-6a4e-46eb-8080-82f904149747/payload-processing/0.log" Nov 03 19:50:32.794034 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:50:32.793942 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:50:57.332137 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:57.332101 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68f76587d6-t8kgt_5feb0c16-6030-4cda-8e9f-243c7a77d023/manager/0.log" Nov 03 19:50:57.856940 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:57.856906 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68f76587d6-t8kgt_5feb0c16-6030-4cda-8e9f-243c7a77d023/manager/0.log" Nov 03 19:50:58.606755 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:58.606727 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kt9k6_53a4c748-6a4e-46eb-8080-82f904149747/payload-processing/0.log" Nov 03 19:50:58.828925 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:58.828893 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kt9k6_53a4c748-6a4e-46eb-8080-82f904149747/payload-processing/0.log" Nov 03 19:50:59.181096 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181054 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjsjt/must-gather-vjl9j"] Nov 03 19:50:59.181576 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181555 4948 cpu_manager.go:414] "containerMap: removing container" podUID="2739f09d-06b4-40c2-a7d6-28a7f511feeb" containerName="maas-api" Nov 03 19:50:59.181675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181577 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2739f09d-06b4-40c2-a7d6-28a7f511feeb" containerName="maas-api" Nov 03 19:50:59.181675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181606 4948 cpu_manager.go:414] "containerMap: removing container" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" Nov 03 19:50:59.181675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181614 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" Nov 03 19:50:59.181675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181627 4948 cpu_manager.go:414] "containerMap: removing container" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" Nov 03 19:50:59.181675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181634 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" Nov 03 19:50:59.181675 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181671 4948 cpu_manager.go:414] "containerMap: removing container" podUID="874aa671-864c-4b9b-a69c-e1b473aa08c2" containerName="payload-pre-processing" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181679 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874aa671-864c-4b9b-a69c-e1b473aa08c2" containerName="payload-pre-processing" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181695 4948 cpu_manager.go:414] "containerMap: removing container" podUID="a1216613-b5bf-4235-afb3-fdaf0d072f1c" containerName="payload-processing" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181717 4948 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1216613-b5bf-4235-afb3-fdaf0d072f1c" containerName="payload-processing" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181841 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="874aa671-864c-4b9b-a69c-e1b473aa08c2" containerName="payload-pre-processing" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181856 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1216613-b5bf-4235-afb3-fdaf0d072f1c" containerName="payload-processing" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181872 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="2739f09d-06b4-40c2-a7d6-28a7f511feeb" containerName="maas-api" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181887 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" Nov 03 19:50:59.182043 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.181897 4948 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cb7268-16ad-4e71-8196-4164ac0f4fad" containerName="istio-proxy" Nov 03 19:50:59.185379 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.185358 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.237801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.237773 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e78ba70-3379-4828-ade2-bb960e0a2ebb-must-gather-output\") pod \"must-gather-vjl9j\" (UID: \"9e78ba70-3379-4828-ade2-bb960e0a2ebb\") " pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.237945 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.237901 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4cj\" (UniqueName: \"kubernetes.io/projected/9e78ba70-3379-4828-ade2-bb960e0a2ebb-kube-api-access-bk4cj\") pod \"must-gather-vjl9j\" (UID: \"9e78ba70-3379-4828-ade2-bb960e0a2ebb\") " pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.240067 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.240050 4948 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tjsjt\"/\"default-dockercfg-89plj\"" Nov 03 19:50:59.291716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.291663 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjsjt\"/\"openshift-service-ca.crt\"" Nov 03 19:50:59.291716 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.291687 4948 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjsjt\"/\"kube-root-ca.crt\"" Nov 03 19:50:59.299488 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.299438 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/must-gather-vjl9j"] Nov 03 19:50:59.338857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.338831 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4cj\" (UniqueName: \"kubernetes.io/projected/9e78ba70-3379-4828-ade2-bb960e0a2ebb-kube-api-access-bk4cj\") pod \"must-gather-vjl9j\" (UID: \"9e78ba70-3379-4828-ade2-bb960e0a2ebb\") " pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.339000 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.338895 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e78ba70-3379-4828-ade2-bb960e0a2ebb-must-gather-output\") pod \"must-gather-vjl9j\" (UID: \"9e78ba70-3379-4828-ade2-bb960e0a2ebb\") " pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.339199 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.339180 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e78ba70-3379-4828-ade2-bb960e0a2ebb-must-gather-output\") pod \"must-gather-vjl9j\" (UID: \"9e78ba70-3379-4828-ade2-bb960e0a2ebb\") " pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.400532 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.400506 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4cj\" (UniqueName: \"kubernetes.io/projected/9e78ba70-3379-4828-ade2-bb960e0a2ebb-kube-api-access-bk4cj\") pod \"must-gather-vjl9j\" (UID: \"9e78ba70-3379-4828-ade2-bb960e0a2ebb\") " pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.518919 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.518868 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/must-gather-vjl9j" Nov 03 19:50:59.894857 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.894831 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/must-gather-vjl9j"] Nov 03 19:50:59.896983 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:50:59.896949 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e78ba70_3379_4828_ade2_bb960e0a2ebb.slice/crio-078f12b7132f62740920b287e070a0fa375f3366c2b613708d4642c7817555f4 WatchSource:0}: Error finding container 078f12b7132f62740920b287e070a0fa375f3366c2b613708d4642c7817555f4: Status 404 returned error can't find the container with id 078f12b7132f62740920b287e070a0fa375f3366c2b613708d4642c7817555f4 Nov 03 19:50:59.899241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:50:59.899222 4948 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:51:00.728302 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:00.728271 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/must-gather-vjl9j" event={"ID":"9e78ba70-3379-4828-ade2-bb960e0a2ebb","Type":"ContainerStarted","Data":"078f12b7132f62740920b287e070a0fa375f3366c2b613708d4642c7817555f4"} Nov 03 19:51:01.740093 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:01.740053 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/must-gather-vjl9j" event={"ID":"9e78ba70-3379-4828-ade2-bb960e0a2ebb","Type":"ContainerStarted","Data":"756411c027af72407eca00fdea1af84b9998162291239ec03338881bed78a5cc"} Nov 03 19:51:01.740485 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:01.740104 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/must-gather-vjl9j" event={"ID":"9e78ba70-3379-4828-ade2-bb960e0a2ebb","Type":"ContainerStarted","Data":"25c8e86e7f7898d310151f0ae252cb6d8e2922a0b5f200c20ca2d50c8966e831"} Nov 03 19:51:01.857232 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:01.857121 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tjsjt/must-gather-vjl9j" podStartSLOduration=1.486865804 podStartE2EDuration="2.857106529s" totalImagesPullingTime="1.370240725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:01.856172109 +0000 UTC m=+4391.339168020" watchObservedRunningTime="2026-11-03 19:51:01.857106529 +0000 UTC m=+4391.340102442" Nov 03 19:51:02.794113 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:51:02.794074 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:51:09.030724 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:09.030638 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t4dcp_8da56f0f-6ab9-4eb3-ad1d-35ce98845ada/global-pull-secret-syncer/0.log" Nov 03 19:51:10.303369 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:10.303339 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7dks9_245116c0-8b38-4fd5-868e-fe39362a2113/connection-checker/0.log" Nov 03 19:51:10.728221 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:10.728189 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-d68qq_4a6b250a-51c5-4415-bd6b-4f1275517fdc/connection-checker/0.log" Nov 03 19:51:11.157731 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:11.156003 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pfqbv_d0f084d2-6540-44ee-ac2b-c8e6df92a050/connection-checker/0.log" Nov 03 19:51:12.771617 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:12.771583 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nl6fg_d0ee858f-ae52-4de9-b576-ed63754072f9/konnectivity-agent/0.log" Nov 03 19:51:13.248612 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:13.248579 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-95.us-west-2.compute.internal_3e31283f871e3314b8eda74a4d6e8f72/haproxy/0.log" Nov 03 19:51:18.113514 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.113480 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pj4wz_7995752d-b5e3-4b41-b7e7-e9893ca31b33/manager/0.log" Nov 03 19:51:18.528780 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.528744 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-rfj7j_3bb8de85-fed2-48d4-aeda-c1e45945aeeb/manager/0.log" Nov 03 19:51:18.717241 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.717208 4948 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf"] Nov 03 19:51:18.722390 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.722362 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.775439 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.775403 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf"] Nov 03 19:51:18.862414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.862324 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qr2\" (UniqueName: \"kubernetes.io/projected/70a7407a-fbf2-4bac-9c76-446138ce51d2-kube-api-access-j5qr2\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.862414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.862368 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-lib-modules\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.862414 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.862388 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-podres\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.862758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.862520 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-proc\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.862758 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.862553 4948 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-sys\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964121 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qr2\" (UniqueName: \"kubernetes.io/projected/70a7407a-fbf2-4bac-9c76-446138ce51d2-kube-api-access-j5qr2\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964159 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964160 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-lib-modules\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964186 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-podres\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964241 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-proc\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964271 4948 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-sys\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964316 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-sys\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964336 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-proc\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964337 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-podres\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:18.964416 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:18.964268 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70a7407a-fbf2-4bac-9c76-446138ce51d2-lib-modules\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:19.025810 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:19.025777 4948 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qr2\" (UniqueName: \"kubernetes.io/projected/70a7407a-fbf2-4bac-9c76-446138ce51d2-kube-api-access-j5qr2\") pod \"perf-node-gather-daemonset-vqlxf\" (UID: \"70a7407a-fbf2-4bac-9c76-446138ce51d2\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:19.034275 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:19.034243 4948 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:19.413430 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:19.413396 4948 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf"] Nov 03 19:51:19.415481 ip-10-0-128-95 kubenswrapper[4948]: W1103 19:51:19.415452 4948 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod70a7407a_fbf2_4bac_9c76_446138ce51d2.slice/crio-1ce50d113fc51af2c26f1ec0108aadece66fbc7a028c5d25b192eed0301b86e2 WatchSource:0}: Error finding container 1ce50d113fc51af2c26f1ec0108aadece66fbc7a028c5d25b192eed0301b86e2: Status 404 returned error can't find the container with id 1ce50d113fc51af2c26f1ec0108aadece66fbc7a028c5d25b192eed0301b86e2 Nov 03 19:51:19.426020 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:19.425993 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2q8j7_d991fd98-c238-4214-b938-daa90d01841f/manager/0.log" Nov 03 19:51:19.886165 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:19.886128 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" event={"ID":"70a7407a-fbf2-4bac-9c76-446138ce51d2","Type":"ContainerStarted","Data":"b611e49592541d76e9d6edf8b07521bae5ccd14d315736f3ceac72fbdb156c32"} Nov 03 19:51:19.886375 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:19.886189 4948 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" event={"ID":"70a7407a-fbf2-4bac-9c76-446138ce51d2","Type":"ContainerStarted","Data":"1ce50d113fc51af2c26f1ec0108aadece66fbc7a028c5d25b192eed0301b86e2"} Nov 03 19:51:19.886375 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:19.886218 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:20.005156 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:20.005079 4948 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" podStartSLOduration=2.005057947 podStartE2EDuration="2.005057947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:20.002900775 +0000 UTC m=+4409.485896700" watchObservedRunningTime="2026-11-03 19:51:20.005057947 +0000 UTC m=+4409.488053867" Nov 03 19:51:20.446019 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:20.445983 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nmztb_be9f4c2f-1332-4a3f-a756-96959733fa78/manager/0.log" Nov 03 19:51:25.906930 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:25.906901 4948 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-vqlxf" Nov 03 19:51:25.976202 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:25.976173 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-75cdf_fd5c02ff-a13f-4d45-bb84-df74ec8fd86f/cluster-monitoring-operator/0.log" Nov 03 19:51:28.571024 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:28.570994 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gv75s_af2bf8da-0ed4-4795-b756-e9955bce76a8/node-exporter/0.log" Nov 03 19:51:28.994801 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:28.994759 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gv75s_af2bf8da-0ed4-4795-b756-e9955bce76a8/kube-rbac-proxy/0.log" Nov 03 19:51:29.407257 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:29.407188 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gv75s_af2bf8da-0ed4-4795-b756-e9955bce76a8/init-textfile/0.log" Nov 03 19:51:32.793680 ip-10-0-128-95 kubenswrapper[4948]: E1103 19:51:32.793633 4948 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g96rr" containerName="extractor" Nov 03 19:51:42.004634 ip-10-0-128-95 kubenswrapper[4948]: I1103 19:51:42.004606 4948 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c7898tnnp_38f23fde-7287-45c5-9e3a-31a0ae356e1d/istio-proxy/0.log"